Soma-notes - User contributions [en]

Talk:COMP 3000 Essay 2 2010 Question 8

2010-12-01T21:48:29Z

Cling: Modesty Fix

Group Members

Trevor Bonesaw Malone - tmalone@connect.carleton.ca //FIRST POST!

Qi Zhang - qzhang13@connect.carleton.ca

Gregory Bint - gbint@connect.carleton.ca

Gautam Akiwate - gakiwate@connect.carleton.ca

Corey Ling - cling@connect.carleton.ca

Sarah Liske

==Relevant Sources==
*NEWSOME,J.,AND SONG,D.Dynamic Taint Analysis for Automatic Detection, Analysis, and Signature Generation of Exploits on Commodity Software. [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.83.2141&rep=rep1&type=pdf Dynamic Taint Analysis for Automatic Detection]
Seems to be THE Dynamic Taint Analysis Paper.Talks about implementation on TaintCheck. Could be also useful for critique section -[Gautam]

== Work Plan ==

As Trevor intimated, we should have clear division of work going forward. This is sort of the break down as I see it. Please edit as you think of new ideas!

* Background Concepts
** Information Flow Theory. (Implicit and Explicit Flows.) --Done[--[[User:Gautam|Gautam]] 03:54, 28 November 2010 (UTC)]
** What is dynamic taint analysis --Done[--[[User:Gautam|Gautam]] 05:07, 28 November 2010 (UTC)]
** What is the difference between dynamic and static analysis - it's there, who added it?
* Research Problem
** How do we build a DTA engine for a phone? - done, but by who?
** Why do we want to? (information misuse) - done, but by who?
* Contribution
** How did they implement their DTA engine (Done: --[[User:Cling|Cling]] 04:50, 26 November 2010 (UTC))
** What did they find about information misuse (Done: --[[User:Cling|Cling]] 04:50, 26 November 2010 (UTC))
** Compared to the existing taint tracking approaches. [[User:Zhangqi|Zhangqi]] 07:11, 27 November 2010 (UTC) (Added something. Still looking for other examples,in progress)
** (What else should be in the contributions? Anything need fleshing out?) (Working on that now :) ) sliske
* Critique
**Added two paragraphs at the end of the present critique. Please incorporate it into your content as you deem fit.--[[User:Gautam|Gautam]] 09:07, 30 November 2010 (UTC)
**^ done. fleshed out critique, and added a bit about how taintdroid doesn't track implicit flow. Also reworded (the entire essay) for clarity where necessary/checked spelling. It would be a good idea for everyone to read it over once for spelling/clarity before thursday, just in case something doesn't make sense - sliske
* References
** The article has 61 references! We can probably use some of them
**whee! reading papers and sticking in information as need be. Also working out how to cite properly, as there are two citations used currently
references added and citations taken care of. will go over fill in a few places where information may be lacking after class sliske

List of information we need to find external sources for:
* History of taint analysis
* History of privacy research relating to smart phones

== Work In Progress ==

Log what you are working on *right now* so that other people don't try to do the same thing. Make sure to clear your name from here when you are done.

* Gregory Bint: Research Problem
** Need to find some history on smart phone security research for the second part.

* Gautam Akiwate: Background Concepts
** Any resources on Dynamic taint Analysis would be appreciated!

* Qi Zhang, Corey Ling: Contributions

* Trevor Malone: Critique

* Sarah Liske: References and Questions, Clarity/Spelling.

== Some Notes from the Video ==

Tracking of privacy sensitive data through Dynamic Taint Analysis (aka. Taint Tracking). The trick is to mark private data as it sourced, and then follow those marks until (unless) they leave the phone.

Android phones run Java apps, which are compiled into DEX, and then run on top of the Dalvik VM. It is this VM that we modify so that we can support the storage and tracking of taint tags.

Taint sources
* low -bandwidth sensors
** Location
** Accelerometer
* High-bandwidth sensors
** Mic
** Camera
* Information DB
** Address book
** SMS storage
* Device ID
** IMEI
** IMSI (don't actually track this one because of false positives)
** ICC_ID
** Phone Number

Taint sink (where marked data can leave the phone)
* Network Taint Sink

Taint propagation
* ???

Taint tags are stored in memory interleaved with the variables they are tracking

Some standard Data Flow technique is used to propagate these tags, especially as one variable that is marked may be assigned to another, so now that variable needs to be tracked as well.

Tracks explicit flows of data, not implicit
To fully capture implicit flows, you need to do static analysis, which is hard with closed-source apps, and cannot be done real-time

Implicit flows are not tracked
* Implicit flows can involve "taint-scope", tracking based on conditionals in code

=== Performance ===

The goal is to create a real time tracking system, so the TaintDroid's performance impact is of some importance

14% CPU overhead
4.4% memory overhead

Macro benchmarks (to get a feel for what the phone's usability is like with TD running)
* App load: 3% (2ms)

=== Findings ===

20 out of 30 tested applications share data in a way that is not expected.

67 of 105 flagged pieces of data leaving the device had no obviously legitimate purpose (verified by the authors).

Many apps sent location data and other unique identifiers to advertising servers.

Most apps do not mention anything to the user.

=== Limitations ===

Tracks only explicit data flows.

An application *could* launder the tags off of the data, if they really wanted to hide this sort of thing from TaintDroid.

There are methods that could be used to protect against this, but they go against the goal of a light-weight, real-time tracking system. TD is not necessarily about catching truly malicious programs, but rather just those that leak information.

Why do apps take this information?
* Lazy; in the demo video, the wallpaper app seems to use the IMEI just as a ready made unique ID
* Overzealous; the developer might thing they *need* the data for something, but actually
* Ads; advertises do seem a little presumptuous in their data collection
* Spying; bosses or spouses
* Malicious;

=== QA Period ===

Q: how do we prevent a malicious app from removing a taint attribute on a file

A: TD operates a too low a level for this to be a problem; TD assumes that the native code is trusted

Q: It seems like you had a lot of false positives

A: The point of this tool was to identify privacy sensitive information as having left the phone, not whether or not a privacy violation has taken place.

Q: Now that TD is released; couldn't malicious apps use some of the methods described in the paper to get around it?

A: Well, yes, but it is not just about maliciousness, it could just laziness or over-zealous ad stuff.

==Other Information==

Hey guys, thought I would just post a generalized paragraph about our essay.

In today’s society, Smartphones are the new big thing. To me that’s what makes this paper so interesting. This paper focuses on private information in android phones and the misuse of this information. The misuse of information includes the SIM card, the ID of the device, or the phone number. TaintDroid is used on smart phones with an efficient taint tracking and analysis system. It has the ability to track sensitive data from multiple sources and examines the misuse of such data. In their study, out of 80 popular third-party applications, TaintDroid monitored that 68 applications had potential misuse of user’s private data. This tool is great for knowing with applications are safe and which are not, so your private data can remained private.

Also, we should really think of splitting up the work in some way. If some people have specific sections they would like to do lets figure that out now so we can divide the workload and get it done over the next couple of days. I don't personally care what part I'm going to have to do, so lets get this going. Any other information people wanna post feel free the more the better, even if we don't end up using it.

[[user:Tmalone|Trevor Malone]]

Hey guys! Anything else we need to get done? Let me know and I can help in anyway possible.

[[user:Tmalone|Trevor Malone]]

Talk:COMP 3000 Essay 2 2010 Question 8

2010-11-26T04:53:46Z

Cling: Feedback Request

Group Members

Trevor Bonesaw Malone - tmalone@connect.carleton.ca //FIRST POST!

Qi Zhang - qzhang13@connect.carleton.ca

Gregory Bint - gbint@connect.carleton.ca

Gautam Akiwate - gakiwate@connect.carleton.ca

Corey Ling - cling@connect.carleton.ca

==Relevant Sources==
*NEWSOME,J.,AND SONG,D.Dynamic Taint Analysis for Automatic Detection, Analysis, and Signature Generation of Exploits on Commodity Software. [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.83.2141&rep=rep1&type=pdf Dynamic Taint Analysis for Automatic Detection]
Seems to be THE Dynamic Taint Analysis Paper.Talks about implementation on TaintCheck. Could be also useful for critique section -[Gautam]

== Work Plan ==

As Trevor intimated, we should have clear division of work going forward. This is sort of the break down as I see it. Please edit as you think of new ideas!

* Background Concepts
** Information Flow Theory. (Implicit and Explicit Flows.)
** What is dynamic taint analysis
** What is the difference between dynamic and static analysis
* Research Problem
** How do we build a DTA engine for a phone?
** Why do we want to? (information misuse)
* Contribution
** How did they implement their DTA engine (Done: --[[User:Cling|Cling]] 04:50, 26 November 2010 (UTC))
** What did they find about information misuse (Done: --[[User:Cling|Cling]] 04:50, 26 November 2010 (UTC))
** (What else should be in the contributions? Anything need fleshing out?)
* Critique
* References
** The article has 61 references! We can probably use some of them

List of information we need to find external sources for:
* History of taint analysis
* History of privacy research relating to smart phones

== Work In Progress ==

Log what you are working on *right now* so that other people don't try to do the same thing. Make sure to clear your name from here when you are done.

* Gregory Bint: Research Problem
** I've got what you will hopefully find to be a reasonable introduction to the problem. I've tried to ask mostly questions here, expecting the details to be covered in Background Concepts and in Contributions.
** I'm going to try to find some history on smart phone security research for the second part.
** Should we move the Research Problem *above* Background Concepts? It might serve as a better lead in that way.

* Gautam Akiwate: Background Concepts
** Any resources on Dynamic taint Analysis would be appreciated!

* Corey Ling: Contributions (Qi Zhang: Original Signed Up)

* Trevor Malone: Critique

== Some Notes from the Video ==

Tracking of privacy sensitive data through Dynamic Taint Analysis (aka. Taint Tracking). The trick is to mark private data as it sourced, and then follow those marks until (unless) they leave the phone.

Android phones run Java apps, which are compiled into DEX, and then run on top of the Dalvik VM. It is this VM that we modify so that we can support the storage and tracking of taint tags.

Taint sources
* low -bandwidth sensors
** Location
** Accelerometer
* High-bandwidth sensors
** Mic
** Camera
* Information DB
** Address book
** SMS storage
* Device ID
** IMEI
** IMSI (don't actually track this one because of false positives)
** ICC_ID
** Phone Number

Taint sink (where marked data can leave the phone)
* Network Taint Sink

Taint propagation
* ???

Taint tags are stored in memory interleaved with the variables they are tracking

Some standard Data Flow technique is used to propagate these tags, especially as one variable that is marked may be assigned to another, so now that variable needs to be tracked as well.

Tracks explicit flows of data, not implicit
To fully capture implicit flows, you need to do static analysis, which is hard with closed-source apps, and cannot be done real-time

Implicit flows are not tracked
* Implicit flows can involve "taint-scope", tracking based on conditionals in code

=== Performance ===

The goal is to create a real time tracking system, so the TaintDroid's performance impact is of some importance

14% CPU overhead
4.4% memory overhead

Macro benchmarks (to get a feel for what the phone's usability is like with TD running)
* App load: 3% (2ms)

=== Findings ===

20 out of 30 tested applications share data in a way that is not expected.

67 of 105 flagged pieces of data leaving the device had no obviously legitimate purpose (verified by the authors).

Many apps sent location data and other unique identifiers to advertising servers.

Most apps do not mention anything to the user.

=== Limitations ===

Tracks only explicit data flows.

An application *could* launder the tags off of the data, if they really wanted to hide this sort of thing from TaintDroid.

There are methods that could be used to protect against this, but they go against the goal of a light-weight, real-time tracking system. TD is not necessarily about catching truly malicious programs, but rather just those that leak information.

Why do apps take this information?
* Lazy; in the demo video, the wallpaper app seems to use the IMEI just as a ready made unique ID
* Overzealous; the developer might thing they *need* the data for something, but actually
* Ads; advertises do seem a little presumptuous in their data collection
* Spying; bosses or spouses
* Malicious;

=== QA Period ===

Q: how do we prevent a malicious app from removing a taint attribute on a file

A: TD operates a too low a level for this to be a problem; TD assumes that the native code is trusted

Q: It seems like you had a lot of false positives

A: The point of this tool was to identify privacy sensitive information as having left the phone, not whether or not a privacy violation has taken place.

Q: Now that TD is released; couldn't malicious apps use some of the methods described in the paper to get around it?

A: Well, yes, but it is not just about maliciousness, it could just laziness or over-zealous ad stuff.

==Other Information==

Hey guys, thought I would just post a generalized paragraph about our essay.

In today’s society, Smartphones are the new big thing. To me that’s what makes this paper so interesting. This paper focuses on private information in android phones and the misuse of this information. The misuse of information includes the SIM card, the ID of the device, or the phone number. TaintDroid is used on smart phones with an efficient taint tracking and analysis system. It has the ability to track sensitive data from multiple sources and examines the misuse of such data. In their study, out of 80 popular third-party applications, TaintDroid monitored that 68 applications had potential misuse of user’s private data. This tool is great for knowing with applications are safe and which are not, so your private data can remained private.

Also, we should really think of splitting up the work in some way. If some people have specific sections they would like to do lets figure that out now so we can divide the workload and get it done over the next couple of days. I don't personally care what part I'm going to have to do, so lets get this going. Any other information people wanna post feel free the more the better, even if we don't end up using it.

[[user:Tmalone|Trevor Malone]]

Talk:COMP 3000 Essay 2 2010 Question 8

2010-11-26T04:50:50Z

Cling: Checklist update

Group Members

Trevor Bonesaw Malone - tmalone@connect.carleton.ca //FIRST POST!

Qi Zhang - qzhang13@connect.carleton.ca

Gregory Bint - gbint@connect.carleton.ca

Gautam Akiwate - gakiwate@connect.carleton.ca

Corey Ling - cling@connect.carleton.ca

==Relevant Sources==
*NEWSOME,J.,AND SONG,D.Dynamic Taint Analysis for Automatic Detection, Analysis, and Signature Generation of Exploits on Commodity Software. [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.83.2141&rep=rep1&type=pdf Dynamic Taint Analysis for Automatic Detection]
Seems to be THE Dynamic Taint Analysis Paper.Talks about implementation on TaintCheck. Could be also useful for critique section -[Gautam]

== Work Plan ==

As Trevor intimated, we should have clear division of work going forward. This is sort of the break down as I see it. Please edit as you think of new ideas!

* Background Concepts
** Information Flow Theory. (Implicit and Explicit Flows.)
** What is dynamic taint analysis
** What is the difference between dynamic and static analysis
* Research Problem
** How do we build a DTA engine for a phone?
** Why do we want to? (information misuse)
* Contribution
** How did they implement their DTA engine (Done: --[[User:Cling|Cling]] 04:50, 26 November 2010 (UTC))
** What did they find about information misuse (Done: --[[User:Cling|Cling]] 04:50, 26 November 2010 (UTC))
* Critique
* References
** The article has 61 references! We can probably use some of them

List of information we need to find external sources for:
* History of taint analysis
* History of privacy research relating to smart phones

== Work In Progress ==

Log what you are working on *right now* so that other people don't try to do the same thing. Make sure to clear your name from here when you are done.

* Gregory Bint: Research Problem
** I've got what you will hopefully find to be a reasonable introduction to the problem. I've tried to ask mostly questions here, expecting the details to be covered in Background Concepts and in Contributions.
** I'm going to try to find some history on smart phone security research for the second part.
** Should we move the Research Problem *above* Background Concepts? It might serve as a better lead in that way.

* Gautam Akiwate: Background Concepts
** Any resources on Dynamic taint Analysis would be appreciated!

* Corey Ling: Contributions (Qi Zhang: Original Signed Up)

* Trevor Malone: Critique

== Some Notes from the Video ==

Tracking of privacy sensitive data through Dynamic Taint Analysis (aka. Taint Tracking). The trick is to mark private data as it sourced, and then follow those marks until (unless) they leave the phone.

Android phones run Java apps, which are compiled into DEX, and then run on top of the Dalvik VM. It is this VM that we modify so that we can support the storage and tracking of taint tags.

Taint sources
* low -bandwidth sensors
** Location
** Accelerometer
* High-bandwidth sensors
** Mic
** Camera
* Information DB
** Address book
** SMS storage
* Device ID
** IMEI
** IMSI (don't actually track this one because of false positives)
** ICC_ID
** Phone Number

Taint sink (where marked data can leave the phone)
* Network Taint Sink

Taint propagation
* ???

Taint tags are stored in memory interleaved with the variables they are tracking

Some standard Data Flow technique is used to propagate these tags, especially as one variable that is marked may be assigned to another, so now that variable needs to be tracked as well.

Tracks explicit flows of data, not implicit
To fully capture implicit flows, you need to do static analysis, which is hard with closed-source apps, and cannot be done real-time

Implicit flows are not tracked
* Implicit flows can involve "taint-scope", tracking based on conditionals in code

=== Performance ===

The goal is to create a real time tracking system, so the TaintDroid's performance impact is of some importance

14% CPU overhead
4.4% memory overhead

Macro benchmarks (to get a feel for what the phone's usability is like with TD running)
* App load: 3% (2ms)

=== Findings ===

20 out of 30 tested applications share data in a way that is not expected.

67 of 105 flagged pieces of data leaving the device had no obviously legitimate purpose (verified by the authors).

Many apps sent location data and other unique identifiers to advertising servers.

Most apps do not mention anything to the user.

=== Limitations ===

Tracks only explicit data flows.

An application *could* launder the tags off of the data, if they really wanted to hide this sort of thing from TaintDroid.

There are methods that could be used to protect against this, but they go against the goal of a light-weight, real-time tracking system. TD is not necessarily about catching truly malicious programs, but rather just those that leak information.

Why do apps take this information?
* Lazy; in the demo video, the wallpaper app seems to use the IMEI just as a ready made unique ID
* Overzealous; the developer might thing they *need* the data for something, but actually
* Ads; advertises do seem a little presumptuous in their data collection
* Spying; bosses or spouses
* Malicious;

=== QA Period ===

Q: how do we prevent a malicious app from removing a taint attribute on a file

A: TD operates a too low a level for this to be a problem; TD assumes that the native code is trusted

Q: It seems like you had a lot of false positives

A: The point of this tool was to identify privacy sensitive information as having left the phone, not whether or not a privacy violation has taken place.

Q: Now that TD is released; couldn't malicious apps use some of the methods described in the paper to get around it?

A: Well, yes, but it is not just about maliciousness, it could just laziness or over-zealous ad stuff.

==Other Information==

Hey guys, thought I would just post a generalized paragraph about our essay.

In today’s society, Smartphones are the new big thing. To me that’s what makes this paper so interesting. This paper focuses on private information in android phones and the misuse of this information. The misuse of information includes the SIM card, the ID of the device, or the phone number. TaintDroid is used on smart phones with an efficient taint tracking and analysis system. It has the ability to track sensitive data from multiple sources and examines the misuse of such data. In their study, out of 80 popular third-party applications, TaintDroid monitored that 68 applications had potential misuse of user’s private data. This tool is great for knowing with applications are safe and which are not, so your private data can remained private.

Also, we should really think of splitting up the work in some way. If some people have specific sections they would like to do lets figure that out now so we can divide the workload and get it done over the next couple of days. I don't personally care what part I'm going to have to do, so lets get this going. Any other information people wanna post feel free the more the better, even if we don't end up using it.

[[user:Tmalone|Trevor Malone]]

Talk:COMP 3000 Essay 2 2010 Question 8

2010-11-26T04:49:01Z

Cling: Added myself to contacts

Group Members

Trevor Bonesaw Malone - tmalone@connect.carleton.ca //FIRST POST!

Qi Zhang - qzhang13@connect.carleton.ca

Gregory Bint - gbint@connect.carleton.ca

Gautam Akiwate - gakiwate@connect.carleton.ca

Corey Ling - cling@connect.carleton.ca

==Relevant Sources==
*NEWSOME,J.,AND SONG,D.Dynamic Taint Analysis for Automatic Detection, Analysis, and Signature Generation of Exploits on Commodity Software. [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.83.2141&rep=rep1&type=pdf Dynamic Taint Analysis for Automatic Detection]
Seems to be THE Dynamic Taint Analysis Paper.Talks about implementation on TaintCheck. Could be also useful for critique section -[Gautam]

== Work Plan ==

As Trevor intimated, we should have clear division of work going forward. This is sort of the break down as I see it. Please edit as you think of new ideas!

* Background Concepts
** Information Flow Theory. (Implicit and Explicit Flows.)
** What is dynamic taint analysis
** What is the difference between dynamic and static analysis
* Research Problem
** How do we build a DTA engine for a phone?
** Why do we want to? (information misuse)
* Contribution
** How did they implement their DTA engine
** What did they find about information misuse
* Critique
* References
** The article has 61 references! We can probably use some of them

List of information we need to find external sources for:
* History of taint analysis
* History of privacy research relating to smart phones

== Work In Progress ==

Log what you are working on *right now* so that other people don't try to do the same thing. Make sure to clear your name from here when you are done.

* Gregory Bint: Research Problem
** I've got what you will hopefully find to be a reasonable introduction to the problem. I've tried to ask mostly questions here, expecting the details to be covered in Background Concepts and in Contributions.
** I'm going to try to find some history on smart phone security research for the second part.
** Should we move the Research Problem *above* Background Concepts? It might serve as a better lead in that way.

* Gautam Akiwate: Background Concepts
** Any resources on Dynamic taint Analysis would be appreciated!

* Corey Ling: Contributions (Qi Zhang: Original Signed Up)

* Trevor Malone: Critique

== Some Notes from the Video ==

Tracking of privacy sensitive data through Dynamic Taint Analysis (aka. Taint Tracking). The trick is to mark private data as it sourced, and then follow those marks until (unless) they leave the phone.

Android phones run Java apps, which are compiled into DEX, and then run on top of the Dalvik VM. It is this VM that we modify so that we can support the storage and tracking of taint tags.

Taint sources
* low -bandwidth sensors
** Location
** Accelerometer
* High-bandwidth sensors
** Mic
** Camera
* Information DB
** Address book
** SMS storage
* Device ID
** IMEI
** IMSI (don't actually track this one because of false positives)
** ICC_ID
** Phone Number

Taint sink (where marked data can leave the phone)
* Network Taint Sink

Taint propagation
* ???

Taint tags are stored in memory interleaved with the variables they are tracking

Some standard Data Flow technique is used to propagate these tags, especially as one variable that is marked may be assigned to another, so now that variable needs to be tracked as well.

Tracks explicit flows of data, not implicit
To fully capture implicit flows, you need to do static analysis, which is hard with closed-source apps, and cannot be done real-time

Implicit flows are not tracked
* Implicit flows can involve "taint-scope", tracking based on conditionals in code

=== Performance ===

The goal is to create a real time tracking system, so the TaintDroid's performance impact is of some importance

14% CPU overhead
4.4% memory overhead

Macro benchmarks (to get a feel for what the phone's usability is like with TD running)
* App load: 3% (2ms)

=== Findings ===

20 out of 30 tested applications share data in a way that is not expected.

67 of 105 flagged pieces of data leaving the device had no obviously legitimate purpose (verified by the authors).

Many apps sent location data and other unique identifiers to advertising servers.

Most apps do not mention anything to the user.

=== Limitations ===

Tracks only explicit data flows.

An application *could* launder the tags off of the data, if they really wanted to hide this sort of thing from TaintDroid.

There are methods that could be used to protect against this, but they go against the goal of a light-weight, real-time tracking system. TD is not necessarily about catching truly malicious programs, but rather just those that leak information.

Why do apps take this information?
* Lazy; in the demo video, the wallpaper app seems to use the IMEI just as a ready made unique ID
* Overzealous; the developer might thing they *need* the data for something, but actually
* Ads; advertises do seem a little presumptuous in their data collection
* Spying; bosses or spouses
* Malicious;

=== QA Period ===

Q: how do we prevent a malicious app from removing a taint attribute on a file

A: TD operates a too low a level for this to be a problem; TD assumes that the native code is trusted

Q: It seems like you had a lot of false positives

A: The point of this tool was to identify privacy sensitive information as having left the phone, not whether or not a privacy violation has taken place.

Q: Now that TD is released; couldn't malicious apps use some of the methods described in the paper to get around it?

A: Well, yes, but it is not just about maliciousness, it could just laziness or over-zealous ad stuff.

==Other Information==

Hey guys, thought I would just post a generalized paragraph about our essay.

In today’s society, Smartphones are the new big thing. To me that’s what makes this paper so interesting. This paper focuses on private information in android phones and the misuse of this information. The misuse of information includes the SIM card, the ID of the device, or the phone number. TaintDroid is used on smart phones with an efficient taint tracking and analysis system. It has the ability to track sensitive data from multiple sources and examines the misuse of such data. In their study, out of 80 popular third-party applications, TaintDroid monitored that 68 applications had potential misuse of user’s private data. This tool is great for knowing with applications are safe and which are not, so your private data can remained private.

Also, we should really think of splitting up the work in some way. If some people have specific sections they would like to do lets figure that out now so we can divide the workload and get it done over the next couple of days. I don't personally care what part I'm going to have to do, so lets get this going. Any other information people wanna post feel free the more the better, even if we don't end up using it.

[[user:Tmalone|Trevor Malone]]

Talk:COMP 3000 Essay 2 2010 Question 8

2010-11-26T04:48:18Z

Cling: Taking my credit till he posts his work

Group Members

Trevor Bonesaw Malone - tmalone@connect.carleton.ca //FIRST POST!

Qi Zhang - qzhang13@connect.carleton.ca

Gregory Bint - gbint@connect.carleton.ca

Gautam Akiwate - gakiwate@connect.carleton.ca

==Relevant Sources==
*NEWSOME,J.,AND SONG,D.Dynamic Taint Analysis for Automatic Detection, Analysis, and Signature Generation of Exploits on Commodity Software. [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.83.2141&rep=rep1&type=pdf Dynamic Taint Analysis for Automatic Detection]
Seems to be THE Dynamic Taint Analysis Paper.Talks about implementation on TaintCheck. Could be also useful for critique section -[Gautam]

== Work Plan ==

As Trevor intimated, we should have clear division of work going forward. This is sort of the break down as I see it. Please edit as you think of new ideas!

* Background Concepts
** Information Flow Theory. (Implicit and Explicit Flows.)
** What is dynamic taint analysis
** What is the difference between dynamic and static analysis
* Research Problem
** How do we build a DTA engine for a phone?
** Why do we want to? (information misuse)
* Contribution
** How did they implement their DTA engine
** What did they find about information misuse
* Critique
* References
** The article has 61 references! We can probably use some of them

List of information we need to find external sources for:
* History of taint analysis
* History of privacy research relating to smart phones

== Work In Progress ==

Log what you are working on *right now* so that other people don't try to do the same thing. Make sure to clear your name from here when you are done.

* Gregory Bint: Research Problem
** I've got what you will hopefully find to be a reasonable introduction to the problem. I've tried to ask mostly questions here, expecting the details to be covered in Background Concepts and in Contributions.
** I'm going to try to find some history on smart phone security research for the second part.
** Should we move the Research Problem *above* Background Concepts? It might serve as a better lead in that way.

* Gautam Akiwate: Background Concepts
** Any resources on Dynamic taint Analysis would be appreciated!

* Corey Ling: Contributions (Qi Zhang: Original Signed Up)

* Trevor Malone: Critique

== Some Notes from the Video ==

Tracking of privacy sensitive data through Dynamic Taint Analysis (aka. Taint Tracking). The trick is to mark private data as it sourced, and then follow those marks until (unless) they leave the phone.

Android phones run Java apps, which are compiled into DEX, and then run on top of the Dalvik VM. It is this VM that we modify so that we can support the storage and tracking of taint tags.

Taint sources
* low -bandwidth sensors
** Location
** Accelerometer
* High-bandwidth sensors
** Mic
** Camera
* Information DB
** Address book
** SMS storage
* Device ID
** IMEI
** IMSI (don't actually track this one because of false positives)
** ICC_ID
** Phone Number

Taint sink (where marked data can leave the phone)
* Network Taint Sink

Taint propagation
* ???

Taint tags are stored in memory interleaved with the variables they are tracking

Some standard Data Flow technique is used to propagate these tags, especially as one variable that is marked may be assigned to another, so now that variable needs to be tracked as well.

Tracks explicit flows of data, not implicit
To fully capture implicit flows, you need to do static analysis, which is hard with closed-source apps, and cannot be done real-time

Implicit flows are not tracked
* Implicit flows can involve "taint-scope", tracking based on conditionals in code

=== Performance ===

The goal is to create a real time tracking system, so the TaintDroid's performance impact is of some importance

14% CPU overhead
4.4% memory overhead

Macro benchmarks (to get a feel for what the phone's usability is like with TD running)
* App load: 3% (2ms)

=== Findings ===

20 out of 30 tested applications share data in a way that is not expected.

67 of 105 flagged pieces of data leaving the device had no obviously legitimate purpose (verified by the authors).

Many apps sent location data and other unique identifiers to advertising servers.

Most apps do not mention anything to the user.

=== Limitations ===

Tracks only explicit data flows.

An application *could* launder the tags off of the data, if they really wanted to hide this sort of thing from TaintDroid.

There are methods that could be used to protect against this, but they go against the goal of a light-weight, real-time tracking system. TD is not necessarily about catching truly malicious programs, but rather just those that leak information.

Why do apps take this information?
* Lazy; in the demo video, the wallpaper app seems to use the IMEI just as a ready made unique ID
* Overzealous; the developer might thing they *need* the data for something, but actually
* Ads; advertises do seem a little presumptuous in their data collection
* Spying; bosses or spouses
* Malicious;

=== QA Period ===

Q: how do we prevent a malicious app from removing a taint attribute on a file

A: TD operates a too low a level for this to be a problem; TD assumes that the native code is trusted

Q: It seems like you had a lot of false positives

A: The point of this tool was to identify privacy sensitive information as having left the phone, not whether or not a privacy violation has taken place.

Q: Now that TD is released; couldn't malicious apps use some of the methods described in the paper to get around it?

A: Well, yes, but it is not just about maliciousness, it could just laziness or over-zealous ad stuff.

==Other Information==

Hey guys, thought I would just post a generalized paragraph about our essay.

In today’s society, Smartphones are the new big thing. To me that’s what makes this paper so interesting. This paper focuses on private information in android phones and the misuse of this information. The misuse of information includes the SIM card, the ID of the device, or the phone number. TaintDroid is used on smart phones with an efficient taint tracking and analysis system. It has the ability to track sensitive data from multiple sources and examines the misuse of such data. In their study, out of 80 popular third-party applications, TaintDroid monitored that 68 applications had potential misuse of user’s private data. This tool is great for knowing with applications are safe and which are not, so your private data can remained private.

Also, we should really think of splitting up the work in some way. If some people have specific sections they would like to do lets figure that out now so we can divide the workload and get it done over the next couple of days. I don't personally care what part I'm going to have to do, so lets get this going. Any other information people wanna post feel free the more the better, even if we don't end up using it.

[[user:Tmalone|Trevor Malone]]

COMP 3000 Essay 2 2010 Question 8

2010-11-26T04:46:06Z

Cling: Added a Contribution Section to the Essay

=Paper=

<big><big>TaintDroid: An Information-Flow Tracking System for Realtime Privacy Monitoring on Smartphones</big></big>

Authors:
* William Enck, ''The Pennsylvania State University''
* Peter Gilbert, ''Duke University''
* Byung-Gon Chun, ''Intel Labs''
* Landon P. Cox, ''Duke University''
* Jaeyeon Jung, ''Intel Labs''
* Patrick McDaniel, ''The Pennsylvania State University''
* Anmol N. Sheth, ''Intel Labs''

Official Website: http://www.appanalysis.org/

Direct Link to Paper: http://appanalysis.org/tdroid10.pdf

Video demonstration of TaintDroid in action: http://www.youtube.com/watch?v=qnLujX1Dw4Y

=Background Concepts=
As a brief glance through the paper might suggest the paper has much to do with "Information Flow Tracking" and "Dynamic Taint Analysis".It is evident that two follow this paper ideas which form the basis of this theory have to be understood. All in all, the following two concepts can be said to be central to understanding this paper. 
*Information Flow
*Taint Analysis
'''<big>Information Flow</big>''' 
Information flow as the name suggests is essentially transfer of information. This transfer of information can be between two processes or withing a given process from let's say a variable x to a variable y. Information Flow Theory tries to quantify this '''flow''' of information into a mathematical model. 
In a security model the Information Flow can be categorized into: 
'''Explicit Flow''' 
Explicit Flow is when information subject to 'security classifications' is transfered to a variable(or process) which is not subject to the same or higher level of 'security' causing a security breach. To put simply explicit flow is when 'secure' information is transferred so that it is publicly observable. 
A Pseudo Code Example: 
<code>PRIVATE VAR <big>secure</big> 
PUBLIC VAR <big>notsecure 
notsecure=secure</big> </code>
In the information in 'secure' which is PRIVATE is transferred to 'notsecure' which is PUBLIC which is an 'Information Leak'.
 
'''Implicit Flow''' 
Implicit Flow is when the information subject to 'security classifications' is deduced indirectly. In this the leakage of information is through the program control flow. Depending on the flow of the program the secure information is compromised. 
A Pseudo Code Example: 
<code>PRIVATE VAR <big>secure</big> 
PUBLIC VAR <big>notsecure 
if secure="blah blah" then: 
insecure=1 
else: 
insecure=0</big> </code>
We can deduce if information in 'secure' is "blah blah" by checking the value of 'insecure'. Information leakage due to implicit flows are much harder to detect and protect from.

For more on Information Flow and its Math Model refer "A Lattice Model of Secure Information Flow" [1]

 '''<big>Taint Analysis</big>'''

Explain briefly the background concepts and ideas that your fellow classmates will need to know first in order to understand your assigned paper.
* Background on Information Flow Theory. Explicit and Implicit Flow.
* Background on the taint data tracking method, how it has been used in other systems (i.e. not phones)
* A reader's digest version of any new articles about this kind of security vulnerability on phones, on apps that collect more personal data than users would expect.

=Research problem=

In today’s society, smartphones are the new big thing. Smartphones, by their nature, are linked into many private details of our lives, including not only classic data like our contact list, but new kinds of data unique to smartphones, such as location data. Except for the odd tunnel or elevator, these phones are constantly connected to the internet. Smartphones also have the ability to download and run third party applications; indeed, this is why we call them "smart". When you combine third party applications with an internet connection, you suddenly find yourself unsure of how your data is being used, that is, what is to stop a third party application from disseminating our private information? As it turns out, very little.

A telling example of this is a wallpaper application that sends your phone number back to the developer. Once the app is running on your phone, it can typically access any of the information on your phone, and it is not necessarily clear when it has done so, or what it is doing with it.

The authors of this paper set out to try to understand what kind of information is being collected and where that information is being sent, and in order to do that, they first needed to build a means of tracking that information.

The strategy they chose is called Dynamic Taint Analysis, sometimes called Taint Tracking. The basic idea being to mark (''taint'') sensitive information at its source, and to then follow that mark as it moves through a system. In the context of this paper, if ever we should see marked data leave the network interface of the phone, then we know that some sensitive information has been disseminated.

There are many difficulties associated with implementing such a system on a smartphone. Their design goals were to create a light-weight, minimal overhead, real-time tracking system that runs directly on a real phone, with real applications. To be really useful, the tracking system must not impact the user experience too heavily.

Some of the difficulties include
* Smart phones are resource constrained. Processing power and memory are limited, and any processing that we do perform will consume battery power. If the tracking system is to be real-time, and for the phone to be considered "usable" by the end user, the system must be truly light weight.
* Third party applications arrive in a compiled format; we cannot analyze their source code.
* Applications may do complex things with the sensitive data. It is unlikely that the application will simply read a location from the GPS and dump it straight out over the network. More likely is that the application will use that data in someway, or combine it with other data, before it is sent. We need to be able to track sensitive data throughout this entire process if we hope to perform any useful analysis.
* Applications can share information with other applications, meaning that our tracking has to work across multiple processes.
* The tracking must operate on a real phone, not a simulated one. With a simulated system, where we control the virtual hardware and memory, we can be certain that we can see everything that an application might do. On a real device, how can we get "low enough" to see everything the applications do?

How does this problem relate to past related work?

=Contribution=
The contributions of the TaintDroid: An Information-Flow Tracking System for Realtime Privacy Monitoring on Smartphones paper is not that they achieved Information flow tracking but that they achieved it efficient enough to run in real time on real constrained hardware devices with minimal overheads. As stated "TaintDroid only incurs an approximate 14% CPU overhead and an approximate 4.4% memory overhead for simultaneously tracking 32 taint markings per data unit." It should also be noted that the 14% CPU over-head is only in regards to a "CPU-bound micro-benchmark and imposes negligible overhead on interactive third-party applications."

This is achieved by modifying the code directly at the VM layer of the Android system to provide variable-level tracking. This allows direct control over how and what private information, such as location details from the GPS, are stored and accessed. Next they modify the JNI layer to provide message-level tracking which allows them to monitor inter-process a.k.a. inter-application communications. This also allows them to "patch the taint propagation on return." so they can keep track of information transfer via native code. Finally modifying the network interface and secondary storage interfaces they are able to provide file-level taint tracking which enables them to ensure "persistent information conservatively retains its taint markings."

By combining these three levels (variable, message and file) of taint tracking, TaintDroid was able to effectively track 30 randomly selected popular 3rd party android applications. In doing so it correctly flagged 105 instances of tainted information transmission. Of these 105, only 35 were legitimate transfers. It also determined that 50% off the applications submitted the users location to advertising servers and 5 of the applications transmitted the users device ID, phone number and SIM card serial number.

From these outstanding numbers you can see that more effective higher granular permission systems are needed and TaintDroid is providing a step in the right direction, by providing a highly efficient real time tracking system.

=Critique=
What is good and not-so-good about this paper? You may discuss both the style and content; be sure to ground your discussion with specific references. Simple assertions that something is good or bad is not enough - you must explain why.

=References=
[1] DENNING, D. E. [http://www.cs.georgetown.edu/~denning/infosec/lattice76.pdf A Lattice Model of Secure Information Flow].
Communications of the ACM 19, 5 (May 1976), 236–243.

Talk:COMP 3000 Essay 2 2010 Question 8

2010-11-26T04:21:56Z

Cling: /* Work In Progress */

Group Members

Trevor Bonesaw Malone - tmalone@connect.carleton.ca //FIRST POST!

Qi Zhang - qzhang13@connect.carleton.ca

Gregory Bint - gbint@connect.carleton.ca

Gautam Akiwate - gakiwate@connect.carleton.ca

==Relevant Sources==
*NEWSOME,J.,AND SONG,D.Dynamic Taint Analysis for Automatic Detection, Analysis, and Signature Generation of Exploits on Commodity Software. [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.83.2141&rep=rep1&type=pdf Dynamic Taint Analysis for Automatic Detection]
Seems to be THE Dynamic Taint Analysis Paper.Talks about implementation on TaintCheck. Could be also useful for critique section -[Gautam]

== Work Plan ==

As Trevor intimated, we should have clear division of work going forward. This is sort of the break down as I see it. Please edit as you think of new ideas!

* Background Concepts
** Information Flow Theory. (Implicit and Explicit Flows.)
** What is dynamic taint analysis
** What is the difference between dynamic and static analysis
* Research Problem
** How do we build a DTA engine for a phone?
** Why do we want to? (information misuse)
* Contribution
** How did they implement their DTA engine
** What did they find about information misuse
* Critique
* References
** The article has 61 references! We can probably use some of them

List of information we need to find external sources for:
* History of taint analysis
* History of privacy research relating to smart phones

== Work In Progress ==

Log what you are working on *right now* so that other people don't try to do the same thing. Make sure to clear your name from here when you are done.

* Gregory Bint: Research Problem
** I've got what you will hopefully find to be a reasonable introduction to the problem. I've tried to ask mostly questions here, expecting the details to be covered in Background Concepts and in Contributions.
** I'm going to try to find some history on smart phone security research for the second part.
** Should we move the Research Problem *above* Background Concepts? It might serve as a better lead in that way.

* Gautam Akiwate: Background Concepts
** Any resources on Dynamic taint Analysis would be appreciated!

* Qi Zhang: Contributions (Corey Ling too apparently)

* Trevor Malone: Critique

== Some Notes from the Video ==

Tracking of privacy sensitive data through Dynamic Taint Analysis (aka. Taint Tracking). The trick is to mark private data as it sourced, and then follow those marks until (unless) they leave the phone.

Android phones run Java apps, which are compiled into DEX, and then run on top of the Dalvik VM. It is this VM that we modify so that we can support the storage and tracking of taint tags.

Taint sources
* low -bandwidth sensors
** Location
** Accelerometer
* High-bandwidth sensors
** Mic
** Camera
* Information DB
** Address book
** SMS storage
* Device ID
** IMEI
** IMSI (don't actually track this one because of false positives)
** ICC_ID
** Phone Number

Taint sink (where marked data can leave the phone)
* Network Taint Sink

Taint propagation
* ???

Taint tags are stored in memory interleaved with the variables they are tracking

Some standard Data Flow technique is used to propagate these tags, especially as one variable that is marked may be assigned to another, so now that variable needs to be tracked as well.

Tracks explicit flows of data, not implicit
To fully capture implicit flows, you need to do static analysis, which is hard with closed-source apps, and cannot be done real-time

Implicit flows are not tracked
* Implicit flows can involve "taint-scope", tracking based on conditionals in code

=== Performance ===

The goal is to create a real time tracking system, so the TaintDroid's performance impact is of some importance

14% CPU overhead
4.4% memory overhead

Macro benchmarks (to get a feel for what the phone's usability is like with TD running)
* App load: 3% (2ms)

=== Findings ===

20 out of 30 tested applications share data in a way that is not expected.

67 of 105 flagged pieces of data leaving the device had no obviously legitimate purpose (verified by the authors).

Many apps sent location data and other unique identifiers to advertising servers.

Most apps do not mention anything to the user.

=== Limitations ===

Tracks only explicit data flows.

An application *could* launder the tags off of the data, if they really wanted to hide this sort of thing from TaintDroid.

There are methods that could be used to protect against this, but they go against the goal of a light-weight, real-time tracking system. TD is not necessarily about catching truly malicious programs, but rather just those that leak information.

Why do apps take this information?
* Lazy; in the demo video, the wallpaper app seems to use the IMEI just as a ready made unique ID
* Overzealous; the developer might thing they *need* the data for something, but actually
* Ads; advertises do seem a little presumptuous in their data collection
* Spying; bosses or spouses
* Malicious;

=== QA Period ===

Q: how do we prevent a malicious app from removing a taint attribute on a file

A: TD operates a too low a level for this to be a problem; TD assumes that the native code is trusted

Q: It seems like you had a lot of false positives

A: The point of this tool was to identify privacy sensitive information as having left the phone, not whether or not a privacy violation has taken place.

Q: Now that TD is released; couldn't malicious apps use some of the methods described in the paper to get around it?

A: Well, yes, but it is not just about maliciousness, it could just laziness or over-zealous ad stuff.

==Other Information==

Hey guys, thought I would just post a generalized paragraph about our essay.

In today’s society, Smartphones are the new big thing. To me that’s what makes this paper so interesting. This paper focuses on private information in android phones and the misuse of this information. The misuse of information includes the SIM card, the ID of the device, or the phone number. TaintDroid is used on smart phones with an efficient taint tracking and analysis system. It has the ability to track sensitive data from multiple sources and examines the misuse of such data. In their study, out of 80 popular third-party applications, TaintDroid monitored that 68 applications had potential misuse of user’s private data. This tool is great for knowing with applications are safe and which are not, so your private data can remained private.

Also, we should really think of splitting up the work in some way. If some people have specific sections they would like to do lets figure that out now so we can divide the workload and get it done over the next couple of days. I don't personally care what part I'm going to have to do, so lets get this going. Any other information people wanna post feel free the more the better, even if we don't end up using it.

[[user:Tmalone|Trevor Malone]]

COMP 3000 Lab 4 2010

2010-11-01T18:01:41Z

Cling: /* Upstart/init */

All of the following should be done with an Ubuntu 10.04 distribution or equivalent. We recommend experimenting in a virtual environment because some of the exercises could make your system unbootable. (In fact, take a snapshot of your working system before starting these exercises so you can easily revert.)

==Questions==

# Change the grub command line at boot to limit the total available RAM to 256M. You'll need to get to select an entry and edit it from within grub.
# Add a new grub menu item which limits the standard kernel to 256M.
# Add a second virtual disk and make it bootable: put the kernel and initial ram disk on it and then install grub. Can you boot off of this disk? What does it do?
# Examine the standard kernel's initial ram disk (initrd). What program is first run in this environment? What does it do?
# Modify the standard initial RAM disk so it pauses for 10 seconds and prints a message to the console on boot.
# What programs does upstart start on boot?

==Hints==

Please add your hints below to help your fellow students!

===Kernel command line options===

===GRUB configuration===

*On Ubuntu the user configuration is stored in /etc/default/grub. The main grub files are stored in /boot/grub. You can update grub's config with the update-grub command.

===How GRUB works===

===Making a disk bootable===

===Examining RAM disks===

*Ubuntu (Debian) store initial RAM disks in the cpio format. 'zcat <file> | cpio -i' will extract its contents.

===Upstart/init===

Upstart "jobs" are config (.conf) files in /etc/init that require one of two options an "exec" line or a "script"

The exec line allows the upstart to just simply execute a script elsewhere, while script allows you to shell script in the upstart job.

A special upstart job is rc.conf which maintains the original runlevel init.d scripts. You will see that rc.conf simply executes all the /etc/init.d scripts.

See more on upstart jobs at http://upstart.ubuntu.com/getting-started.html

Talk:COMP 3000 Essay 1 2010 Question 1

2010-10-14T23:13:28Z

Cling: /* The Essay */

== Microkernel ==
* Moving kernel functionality into processes contained in user space, e.g. file systems, drivers
* Keep basic functionality in kernel to handle sharing of resources
* Separation allows for manageability and security, corruption in one does not necessarily cause failure in system
* Large amount of moving from a process to Kernel to user space and back again, this is a costly operation.

----

''' Microkernel '''
* try's to minimize the amount of software that is mandatory or required [7]
advantages of Microkernel
* favors a modular system structure [7]
* one failure of a program does not impact any other programs [7]
* can support more than one api or strategies since all programs are separated [7]
==== Microkernel Concepts ====
* piece of code is allowed in the kernel only if moving it outside the kernel would adversely affect the system. [7]
* any subsystem program created must be independent of all other subsystem's, any subsystem that is used can guarantee this from all other subsystems [7]
===== Address Space =====
* a mapping that relates the physical page to the virtual page. [7]
* processor specific [7]
* hide's the hardware's concept of address space [7]
* based off the idea of recursion each subsystem has it's own address space [7]
* the micro kernel provides 3 operations [7]
** Grant [7]
*** allows the owner to give a page to a recipient, provided the recipient want's it the page is removed from the owner's address space and but in the recipients. [7]
*** must be available to the owner. [7]
** Map [7]
*** allows the user to share a page with a recipient [7]
*** page is not removed from the owner's address space. [7]
** Flush [7]
*** remove's the page from all recipients address space [7]
*** how does this work with Grant --[[User:Asoknack|Asoknack]] 19:10, 12 October 2010 (UTC)
* allows memory management and paging out side the kernel
* Map and flush is required for memory manger's and pagers [7]
* can be used to implement access right's [7]
* controlling I/O Right's and driver's are not done at kernel level [7]

===== Thread's IPC =====
* Threads
** in the kernel [7]
** Since a thread has an address space , all changes to the thread need to be done by the kernel [7]
* IPC [7]
** in the kernel IPC
** grant and map also need IPC (So buye the priciple above this has to be in the kernel)[7]
** basic way for sub process to communicate. [7]
* Interrupts
** partially in the kernel [7]
** hard ware is a set of thread's which are empty except for there unique sender id [7]
** transformation of the message to the interrupt is done in the kernel [7]
** the kernel is not involved in device - specific interrupt's and does not understand the interrupt. [7]
*** resting the interrupt is done at user level [7]
** if a privileged command is need it is done implicitly the next time an IPC command is sent from the device [7]

===== Unique Identifiers =====

== Virtual Machine ==
* Partitioning or virtualizing resources among OS virtualization running on top of host OS
* Virtualized OS believe running on full machine on its own

----
System Level Virtualization

=== VMM ===
* stands for Virtual Machine Monitor, also known as the hyper-visor[4]
* responsible for virtualization of hardware(mapping physical to virtual) and the VM that run on top of the virtuallized hardware [4]
* usually a small os with no drivers , so it is coupled with a linux distro that provides device / hardware access [4]
** the os that the VMM is using for driver's is called the hostOS [6]
*the hostOS provides login and physical access to the hardware as well as management for the VMM [6]
=== VM ===
* the OS that the vm is running is called the guestOS [6]
* the guestOS only sees resources that have been allocated to the VM [6]
==== three approaches ====
*Type I virtualization [5]
** runs off the physical hardware [4]
** Isolation of the guestOs from the hardware is done threw processe level protection meachnism[6]
*** ring 0 = VMM [6]
*** ring 1 = VM [6]
*** this means all instructions from the VM must go threw the VMM [6]
** since there can be multiple VM's on a computer the scheduling is done by the VMM [6]
** on boot the VMM creates a hardware platform for the VM [6]
** load's the VM kernel into virtual memory and then boot's it like a regular computer [6]
** ex. Xen [4]
*Type II virtualization [5]
** run off the host Os [4]
** ex. VMware , QEMU [4]
* Para-virtualization [6]
** Similar to Type but use the HostOs for Device driver access [6]
** Provide a virtualization that is similar to hardware [From the paper posted, no citation yet]
** GuestOS and Hypervisor work together to improve performance

----
==== ====
(Not complete but most of article 9)
Classical Virtualization
* VMMs allow programs in virtual environments to run natively other than resource usage
** Dominant instructions executed directly on cpu
** vmm completely controls system resources
** often need to emulate every native instruction which would severely effect the performance
** sensitive instruction that violate safety and encapsulation
** vmm handles them as priviledged instructions

x86 Virtualization
* virtualization in personal work stations rather than mainframes
** rings that allow isolation between virtual machines
** most privileged in ring 0 and least in ring 3. The operating system runs in ring 0 and user apps in ring 3
*** vmm in ring 0 and vms in lesser privilege rings (1 or 3)
*** guestOS believes its in ring 0
* address space compression, where to run the VMM
** if run using guest address space, guest can find out its virtualized or compromise the isolation
* does not trap all sensitive instructions but can handle them, violates classical virtualization description
* some privileged access fail without faulting
* interrupt virtualization - VMM handles AND guestOS handles
* binary translation - improve performance
* rewriting instructions and trapping before problems arrise

Paravirtualization
* guestOS become exposed to vm information so that the guest is aware that it is virtualized and can make decisions based on this
* allows to avoid problem instructions
* Xen
* guestOS must be modified and is not natively running
**works with the hostOS to run efficiently

VMM types
* hostedVMM - executes in hostOS and uses the drivers and support of the OS
* Stand-aloneVMM - runs directly on hardware and uses it's own drivers and services
* hybridVMM - runs a serviceOS where requests to hardware go through (I/O)

Device Emulation
* implement real hardware in software
* completely virtual device that the guest interacts with
* mapped to physical hardware that handles the interactions but the emulation allows conversion
* allows the vm to be easily migrated between machines as it does not rely on the physical hardware
* allows having multiple vms and simplifies sharing (multiplexing)
* poor performance as the vmm needs to do a lot to virtulize the machine

Paravirtualization
* modified guestOS to cooperate with VMM
* VMM does not have to do everything to handle device drivers
* not everything can be paravirtualized
* proprietary os and device drivers can't be paravirtualized
* still allows an increase in performance
* eventing or callback mechanism
** guestOS modifies interrupt mechs
* modifications are not applicable to all guestOS

Dedicated Devices
* does not virtualize device but assigns directly to guest vm
* uses guest's drivers instead of host
* simplifiest vmm by removing handing of i/o securily
* limited physical devices that can be dedicated
* dificult to migrate vm as it depends on the pairing with this resource
* elims over-head of virtualization and simplicity in vmm
* direct memory access not supported

== Exokernel ==
* Micro-kernel architecture with limited abstractions, ask for resource, get resource not resource abstraction
* Less functionality provided by kernel, security and handling of resource sharing
* Once application receives resource, it can use it as it wishes/in control
* Keep the basic kernel to handle allocating resources and sharing rather than developing straight to the hardware

----
* multiplex resources securely providing protection to mutual distrustful application threw the use of secure binding's[1]
* Goal of the exokernel is to give LibOS maximum freedom with out allowing them to interfere with each other. to do this the exokernel separates protection from management in doing this it provide 3 important tasks[1]
** tracking ownership of resources [1]
** ensuring protection by guarding all resource usage and binding points (not to shure what binding points are)[1]
** revoking access to the resources [1]
* LibrayOS (LibOs)
** Reduces the number of kernel crossings[1]
** Not trusted by the exokernel so can be trusted by the application , Example given is a bad parameter passed to the LibOs only the application is affected.[1] (So LibOs cant interact with kernel ???)
** Any application running on the Exokernel can change the LibrayOs freely [1]
** Application that use LibOS that implement standard interfaces (POSIX) will be portable on any system with the same interface [1]
** LibOs can be made portable if it is designed to interact with a low-level machine independent level to hide hardware details [1]

=== Exokernel Design ===
==== Design Principles ====
*Securely Expose Hardware [1]
** an Exokernel tries to create low level primitives that the hardware resources can be accessed from, this also includes interrupts,exceptions [1]
** the exokernel also export privileged instructions to the LibOS so that traditional OS abstractions can be implemented (eg Process , address pace)[1]
** Exokernels should avoid resource management except when required protection ( allocation , revocation , ownership)[1]
** application based resource management is the best way to build flexible efficient flexible systems [1]
*Expose allocation[1]
** allow LibOs to request physical resources [1]
** resource allocation should not be automatic, the LibOS should participate in every single allocation decision [1]
*Expose Names[1]
** Use physical name's when ever possible[3] (not to sure what physical names are, I think it is as simple as what the hardware is called)--[[User:Asoknack|Asoknack]] 20:27, 9 October 2010 (UTC)
** Physical names capture useful information [3]
*** safer than and less resource intensive than virtual names as no translations are needed[3]
*Expose Revocation [1]
** use visible revocation protocol [1]
** allows well behaved LibOS to preform application level resource management [1]
** Visible revocation allows the LibOS to choose what instance of the resource to release[1](Visible means that when revocation happens the exokernel tell the LibOS that resource is being revoked)
''' Policy '''
* LibOS handle resource policy decisions
* Exokernels have a policy to decided between competing LibOS (Priority , share of resources)
** it enforces this threw allocation and deallocation (every thing can achieved threw this even what block to write and such)

==== Secure Bindings ====
* Used by the exokernel to allow the LibOS to bind to resources [1]
* Allows the separation of protection and resource use [1]
* only checks authorization during bind time [1]
** Application's with complex needs for resources only authorized during bind.[1]
* access checking is done during access time and there is no need to understand complex resources needs during access[1]
** (this means that the exokernel checks once to make sure an application has authorization once approved, when the application tries to use the resource the exokernel is only concerned about policy conflict's)--[[User:Asoknack|Asoknack]] 18:20, 9 October 2010 (UTC)
** allows the kernel to protect the resources with out understanding what the resource is [1]
*three way's to implement
* Hardware Mechanisms [1]
* Software caching [1]
* Downloading application code [1]
''' Downloading Code to the Kernel '''
* used to implement secure bindings , and improve performance[1]
** eliminate the number of kernel crossings [1]
** downloaded code can be run with out the application to be scheduled [2]
==== Visible Resource Revocation ====
* Used for most resources [1]
** allows for LibOS to help with deallocation [1]
** LibOS are able to garner what resources are scare [1]
* Slower than Invisible as application involvement is required [1]
** ex of when invisible is used is Processor addressing-context identifiers [1]
==== Abort Protocol ====
* allows the exokernel to take resources away from the LibOS [1]
* used when the LibOS fails to respond to the revocation request [1]
* Exokernel must be careful not to delete as the LibOS might need to write some system critical data to the resource [1]

== Comparisons ==
====Exokernel/Microkernel====
'''Similarities'''
* Limited functionality in kernel
** functionality in kernel to handle sharing of resources and security
** avoids programming directly to hardware which creates a dependency
* Additional functionality provided in user space as processes
'''Differences'''
* Minimal abstractions provided by the kernel
** Applications given more power in exokernel

====Exokernel/VM====
'''Similarities'''
* Idea of partitioning resources between applications/OSs
* "Control" of resource given
* Isolation from other applications/OSs
'''Differences'''
* Exokernel runs applications, VM runs OS
* VM uses a hostOS and guestOSs run on top
* Virtualization on VMs, Exokernel deals with real resources
* VM hides a lot of information because it emulates. Exokernel does not.

====Microkernel/VM====
'''Differences'''
* With a virtual machine, you are not virtualizing apps like with a microkernel but virtualizing an entire Operating System.
* This can be costly but the benefits are that it's easier and all the standard OS features are available.

== References ==
[1]<nowiki> Engler, D. R., Kaashoek, M. F., and O'Toole, J. 1995. Exokernel: an operating system architecture for application-level resource management. In Proceedings of the Fifteenth ACM Symposium on Operating Systems Principles (Copper Mountain, Colorado, United States, December 03 - 06, 1995). M. B. Jones, Ed. SOSP '95. ACM, New York, NY, 251-266. DOI= http://doi.acm.org/10.1145/224056.224076 </nowiki>

[2]<nowiki>Engler, Dawson R. "The Exokernel Operating System Architecture." Diss. Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 1998. Web. 9 Oct. 2010. <http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.61.5054&rep=rep1&type=pdf>.</nowiki>

[3]<nowiki>Kaashoek, M. F., Engler, D. R., Ganger, G. R., Briceño, H. M., Hunt, R., Mazières, D., Pinckney, T., Grimm, R., Jannotti, J., and Mackenzie, K. 1997. Application performance and flexibility on exokernel systems. In Proceedings of the Sixteenth ACM Symposium on Operating Systems Principles (Saint Malo, France, October 05 - 08, 1997). W. M. Waite, Ed. SOSP '97. ACM, New York, NY, 52-65. DOI= http://doi.acm.org/10.1145/268998.266644 </nowiki>

[4]<nowiki>Vallee, G.; Naughton, T.; Engelmann, C.; Hong Ong; Scott, S.L.; , "System-Level Virtualization for High Performance Computing," Parallel, Distributed and Network-Based Processing, 2008. PDP 2008. 16th Euromicro Conference on , vol., no., pp.636-643, 13-15 Feb. 2008
DOI= http://doi.acm.org/10.1109/PDP.2008.85 </nowiki>

[5]<nowiki>Goldberg, R. P. 1973. Architecture of virtual machines. In Proceedings of the Workshop on Virtual Computer Systems (Cambridge, Massachusetts, United States, March 26 - 27, 1973). ACM, New York, NY, 74-112. DOI= http://doi.acm.org/10.1145/800122.803950 </nowiki>

[6]<nowiki>Vallee, G., Naughton, T., and Scott, S. L. 2007. System management software for virtual environments. In Proceedings of the 4th international Conference on Computing Frontiers (Ischia, Italy, May 07 - 09, 2007). CF '07. ACM, New York, NY, 153-160. DOI= http://doi.acm.org/10.1145/1242531.1242555 </nowiki>

[7]<nowiki>Liedtke, J. 1995. On micro-kernel construction. In Proceedings of the Fifteenth ACM Symposium on Operating Systems Principles (Copper Mountain, Colorado, United States, December 03 - 06, 1995). M. B. Jones, Ed. SOSP '95. ACM, New York, NY, 237-250. DOI= http://doi.acm.org/10.1145/224056.224075 </nowiki>

[8]<nowiki>Microkernel verses monolithic kernel
http://www.vmars.tuwien.ac.at/courses/akti12/journal/04ss/article_04ss_Roch.pdf - Roch</nowiki>

I will site it/reference it better later

[9]Fisher-Ogden J. 2006. Hardware Support for Efficient Virtualization. University of California, San Diego. http://cseweb.ucsd.edu/~jfisherogden/hardwareVirt.pdf

Not completely sure of the citation style used above.

== Unsorted ==
An overview of exokernels,virtual machines, microkernels *[http://www2.supchurch.org:10999/files/school/classes/CSCI4730/Lectures/grad-structures.ppt Overview](Power Point) 
Should not be used as a source but an overview.

The original paper on [http://portal.acm.org/citation.cfm?id=224076 Exokernels] --[[User:Gautam|Gautam]] 22:39, 6 October 2010 (UTC)

Exokernel-
Minimalistic abstractions for developers
Exokernels can be seen as a good compromise between virtual machines and microkernels in the sense that exokernels can give that low level access to developers similar to direct access through a protected layer and at the same time can contain enough hardware abstraction to allow similar benefit of hiding the hardware resources to application programs.
Exokernel – fewest hardware abstractions to developer
Microkernel - is the near-minimum amount of software that can provide the mechanisms needed to implement an operating system
Virtual machine is a simulation of any or devices requested by an application program
Exokenel – I’ve got a sound card
Virtual Machine – I’ve got the sound card you’re looking for, perfect virtual match
Microkernel – I’ve got sound card that plays Khazikstan sound format only
MicroKernel - Very small, very predictable, good for schedualing (QNX is a microkernel - POSIX compatable, benefits of running linux software like modern browsers)

This is some ideas I've got on this question, please contribute below
-Rovic

Outlining some main features here as I see them.

I found that the exokernel was an even lower-level design than the microkernel, closer to the hardware without abstraction. They have the same architecture with the basic functionality contained in the kernel to manage everyone. As the exokernel "gives" the resource to the application it can use the resource in isolation of other applications (until forced to shared) much like VMs receive their resources, either partitioned or virtualized, and execute as if its running on its own machine. There is this similar notion of partitioning the resources among applications/OS and allowing them to take control of what they have.

I'll locate some references later on. --[[User:Slay|Slay]] 15:00, 7 October 2010 (UTC)

I'm just going to post my answer for question 1 on the individuel assignment and hope it helps. --[[User:Aellebla|Aellebla]] 15:06, 12 October 2010 (UTC)

The design of the micro kernel was to take everything they could out of the Kernel and put it into a process. For ex, networking would be put into a process instead of staying in the kernel. The micro kernel dev's tried to keep lots of things in user space for efficiency. But one major problem with this is there would be a large amount of moving from a process to the kernel to user space and back again and this is a costly, non efficient process.It was an application specific OS, there was no multiplexing. With a virtual machine you are not virtualizing apps like with a microkernel but virtualizing an entire Operating System. This is very heavy however but the benefits are that it‟s easy and all the standard OS features are there whereas in a microkernel setup they would not all be there and this can be seen as a compromise.

Exokernels can be seen as a compromise to virtual machines and microkernels because virtual machines emulate and exokernels do not. When you emulate something you hide a lot of the actual information because you wouldn‟t be able to see the „real‟ hardware. If we look at a virtual box setup running Linux, and we go look at all the hardware, it will be displayed as fake hardware.

Maybe we can have an introduction - paragraph or so on each type - then similarities - differences - and the compromise. I am going to do some research and writing this weekend and I will put some up -- Jslonosky

btw in my page (i guess you can call it that) i have some resources i have found --[[User:Asoknack|Asoknack]] 15:50, 8 October 2010 (UTC)
- Wow, nice man. I will go ahead and write up the descriptive paragraphs on each kernel and virtual machine if no one minds. --Jslonosky

I think we should divide up the paragraphs and proofread each others instead. (Are there only 4 of us?) I don't have much time to work on this today though but I'll try to work on it tomorrow morning. - Slay

Sure guy. That sounds good. There should be 5 or 6 of us though.. . Oh well. Their loss. I will do some before or after work today. Ill start with Microkernel since there is not a large amount of info here, and so we don't overlap each other - JSlonosky

yeah i think there was more like 7 of us btw if any one has any more information feel free to add it would be nice if you add the references so that way citing is really easy on acm.org it will auto give you the citation info (where it says Display Formats click on ACM Ref and new window with the citation info auto pop's up) --[[User:Asoknack|Asoknack]] 02:28, 11 October 2010 (UTC)

I added an outline of the similarities and differences. Add any more that I missed. These are from observations so I don't have any resources. -Slay
That's probably fine. Our textbook probably outlines some of them, so I am sure we can find a few there - JSlonosky

Talked to the teacher today and for VM he said we should focus on the implementation such as Xen and VMware , he also said to talk about para virtualization --[[User:Asoknack|Asoknack]] 18:42, 12 October 2010 (UTC)

A paper about emulation and paravirtualization [http://portal.acm.org/citation.cfm?id=1189289&coll=GUIDE&dl=GUIDE&CFID=105648137&CFTOKEN=47153176&ret=1#Fulltext link] - Slay

Oh no big words. Sorry about the Microkernels not done yet. Working on an outline now. Finally found how to access the ACM through carleton. Gawd.
I am planning an outline, quick bit about kernels in general, (maybe mention monolith kernels?), and what microkernels do.
I see the microkernel outline info and a reference ( Whomever did that == hero: true) about the scheduling and the Memory management. Should that be included in kernels in general and then mention what microkernels build upon/change? - JSlonosky

Sorry late to the party here. My mistake was not checking the discussion page when I checked in. I don't want to trample anyone's current work but I don't see any work on the final essay done. I would love to help just need to know where I can step in so as to not screw anyone else up. -- [[User:Cling|Cling]]

I don't think I'll be able to write up something for the final essay, even though I suggested splitting it. I'll do research tonight though on the paravirtualization. If I find the time, I'll try to write something. Sorry about that. --[[User:Slay|Slay]] 21:52, 13 October 2010 (UTC)

We all have 3004 to do too, man. I do not think anyone has chosen to do Virtual Machine section yet, or the Exokernel itself. But the contrast paragraph and the intro is chosen, and intro is done. Microkernel and kernel will be done in a hour I hope. -- JSlonosky

I can attempt to write up anything, the issue is I don't have any context on what to write, how do I tie it in to the rest of the essay? I only have a Japanese Quiz tomorrow morning then I should be good to write anything up for the rest of the day. As someone who has already written part of the essay, and assuming I attempt the exokernel section, how much do you think I should write? Should it just be about exokernel or should there be comparisons to the other topics? Thanks --[[User:Cling|Cling]] 23:14, 13 October 2010 (UTC)

Go with the Exokernel itself. Slade is getting off work in a hour and we can double check what he is doing then. We can put it together tomorrow sometime, and fill in the other stuff. - JSLonosky

I'll attempt to work on VM tonight, then. I would feel so bad if I didn't write anything. -Slay

Still wondering how much to write, I think we should decide on a decent word count or length so we don't have one short section (which would probably be mine) and/or one massive section that dwarfs all the others. If anyone has already written a section could you post your word count so we can aim to be around there, it would obviously be just a recommendation but it's just better to be on the safe side and have everything uniform. I haven't seen any formal requirements for the essay but I could be wrong, I also haven't been to class in a while. --[[User:Cling|Cling]] 23:33, 13 October 2010 (UTC)

Yeah Slay, VM probably doesnt have much to write about. Get something down, and we can go over it. CLing, Just write what you think. There is not a lot to go over if I write kernel/microkernel well enough. What is a exokernel? exokernel was an even lower-level design than the microkernel, closer to the hardware without abstraction, basically (As said by Slade). I will probably end up with 500 or a bit more words. -- JSlonosky

Sound off!

Who's actually reading this? Add your name to the list...

Rovic P.
Jon Slonosky
Corey Ling
Steph Lay
Aaron .L

== The Essay ==

Let's actually breakdown the essay into components then write it here.

I'd like to go along the premise that microkernels and and virtual machines are "weaker" than exokernels in design for the essay. If anyone has any objections, add it here.

-Slade

what do you mean by "weaker"(i think you mean exokernels' takes the best of both worlds ) --[[User:Asoknack|Asoknack]] 02:45, 13 October 2010 (UTC)

What I mean by weaker is that we should focus on the things microkernels and virtual machines may not do as well compared to a system based off an exokernel design and then focus on how an exokenenel can take the best of both worlds. Please choose which section you will work on, that's not to say it'll be the only part you do, but rather we'll all contribute to each part please. 1 day left.
-Slade

...to the extent that exokernels be seen as a compromise between virtual machines and microkernels.
-I'll work on the initial intro. -Slade

3 paragraphs that prove it
Explain how the key design characteristics of these three system architectures compare with each other.

intro/thesis statement -Rovic P.

In Computer Science, the kernel is the component at the center of the majority of operating systems. The kernel is a bridge for applications to access the hardware level. It is responsible for managing the system's resources such as memory, disk storage, task management and networking. We are comparing Exokernels to Microkernels and Virtual Machines by looking at how the kernel goes about such management and its connections. In the Exokernel conceptual model, we can see exokernels become much smaller than microkernels because as this design shows, they are tiny and strive to keep functionality limited to protection and multiplexing of resources. The Virtual Machine Implementation of virtualizing all devices on the system may provide compatibility, but it also adds a layer of complexity within the system. This is less efficient than a real machine as it accesses the hardware indirectly. It can be observed by examining how the exokernel provides low level hardware access and provides custom abstraction to those devices. This is done in order to improve program performance as opposed to a VM's implementation. The exokernel concept has a design that can take the better concepts of microkernels and virtual machines to the extent that exokernels can be seen as a compromise between a virtual machine and a microkernel.

Paragraph 1 -Microkernel -Jon S.

The kernel is the most important part of an operating system. An operating system could not function without the kernel.

A kernel is the lowest level section of an operating system. Within a system, it has the most privileges. It runs along side of the ‘user space’. It is in the ‘user space’ where a user has access. This is also where the user can run its applications and libraries.[8] This leaves the kernel with the need to manage the other necessary processes. For example, the kernel could manage the File Systems and complete process scheduling. The kernel is layered with the most authoritative process on its lowest level.[8] A monolithic kernel, which is a kernel that contains all mandatory processes within itself, was the common kernel type of the earlier versions of today’s operating systems utilized. However, this architecture had problems. [8] If the kernel needed to be updated with more code, or a change in the system, the entire kernel would need to be compiled. Therefore, due to the amount of processes within it, it would take an inefficient amount of time. Here, a microkernel becomes practical.

The concept of a microkernel, is to reduce the code within the kernel. The microkernel is only included in the kernel if it would impact the system. There are a variety of ways the system could be affected if a microkernel were to be implemented, for example, there would be increased performance and efficiency. [7] Furthermore, a microkernel is a kernel that has a reduced amount of mandatory software within itself. This means that it contains less software to manage, and has a reduced size.

A microkernel that emerged at the end of the 1980’s to the early 1990’s has the structure that processes as if the File Systems and the Drivers are removed from it, leaving the kernel with process control and input/out control, and interrupts. [8] This new structure makes the system much more modular, and easier to provide solutions. If a driver must be patched or upgraded, the kernel does not need to be recompiled. [7] The old driver can be removed, and during the time the device waits for the system to recognize it, the operating system replaces the driver. This allows for real-time updating, that can be done while the computer is still functional. This can reduce the complete crash of the system. Therefore,if a device fails, the kernel will not crash itself, like a monolithic kernel would. The microkernel can reload the driver of the device that failed and continue functioning. [7]

Want more on the scheduling? I can do that if wanted. -key note on exokernel's mutiplexing vs microkernel's messaging, exo more efficient so perhaps running with the idea that messaging b/w processes not necessarily the ideal way need to also start outlaying weaknesses in the design as well in order to play up the idea that an exokernel just does it better -Slade

Paragraph 2 -Virtual Machine -Steph L.

A Virtual Machine, or VM, is a software abstraction of a physical machine. This entails virtualization of the physical machines resources in order to share them among OS run in the VM. Virtualizing these resources allow the OS to run as if it were on a full machine when, in reality, it is actually running in a virtualized environment on top of a hostOS. The OS is actually running on the machine, sharing the resources.

Virtual Machines generally contain two key components; both the Virtual Machine Monitor, or VMM and the VM.

The VMM, also known as the hypervisor, manages the virtualization of the physical resources and the interactions with the VM running on top. [4] In other words, it mediates between the virtualized world and the physical world, keeping them separate and monitoring their interactions with each other. The hypervisor is what allows the VM to operate as if it were on its' own machine. This is done by handling any requests to resources and maintaining these requests with what has actually been provided to the VM, by the hostOS. The hostOS provides management for the VMM, as well as, allowing physical access to devices, hardware and drivers. [6]

The VM, is what contains the OS, we are running through virtualization. [6] This OS is called the guestOS. It will only be able to access any resources that have been made available to the VM by the hostOS. [6] Otherwise, the guestOS will not know about any other resources and does not have direct access to physical hardware. This will be taken care of by the VMM. While, the guestOS will execute as its own machine, unaware of this mediator.

There are various ways of implementing hardware virtualization in a system to allow VMs to run. This includes device emulation, paravirtualization and dedicated devices. [9]

In device emulation the VMM provides a complete virtualization of a device for the guestOS to interact with, in the software. [9] The VMM will map this virtualized device to the physical resource and handle any interactions between them. This will usually include converting instructions from the guestOS into instructions that are compatible with the device. [9] Device emulation allows for the VM to be migrated easily to another machine as it is not dependent on the physical devices but rather, on the software emulations instead. [9] It also allows for simpler multiplexing between multiple virtual machines as it can handle sharing though these virtualized devices. [9] A drawback of emulation, however, is poor performance. This is because the VMM must handle every request and convert them to be compatible with the physical device. [9] Nonetheless, despite its poor performance, emulation is still the most common form of virtualization.

Paravirtualization allows for a boost in performance by having the guestOS and the hostOS work together to improve performance. [9] In paravirtualization, the guestOS is not a native OS and must be modified so guestOS is aware that it is a virtualized system. [9] Since the guestOS is aware of this, it can now make better decisions about how it accesses devices. Seeing as the guestOS will be able to handle its decisions better, the VMM’s responsibility is reduced as it now does not have to translate between the guestOS and the physical devices. [9] Though the performance boost is a great advantage, there are many disadvantages to this. Some disadvantages are that you can only use paravirtualization if you can implement the modifications to the guestOS. As well, not everything can be paravirtualized and as such, this limits the cases in which this method can be used. [9] Also, every guestOS must be modified in order to be used in paravirtualization. The modifications will differ in various OS and so, there is also the task of implementing these changes to make a guestOS compatible. [9]

Instead of virtualizing the hardware and mediating between the VMM, dedicated devices allow mapping directly to the guestOS. [9] In this method, the device will use the guestOS’s drivers instead of the hostOS’s. [9] Using this method allows the guestOS to use the hardware to its full extent, without having to deal with the VMM. This then, simplifies the VMM by eliminating the overhead by virtualizating the hardware and handling the requests to devices. [9] However, there are limited physical resources to be dedicated to a guestOS. This also makes migration difficult as the guestOS is dependent on the physical device. [9]

'''notes'''
- it ended up being quite lengthy. I mainly focused on the device virtualization rather than the architecture of a VM (like x86 virtualization). I'll put up my notes for the paper I found for virtualization. I didn't talk about Xen or VMware though. If any of that is needed, I can try to continue working on it tonight but I have another priority.

-try focusing on the emulation side of VM where emulation's weaknesses vs direct hardware access or custom abstraction that exokernels -Slade

Paragraph 3 -Exokernel -Corey L

Exokernels can be seen as a compromise between a microkernel and a VM. It can also be seen as simply dividing a monolithic kernel up into 2 parts. The management tasks of the kernel remain in the exokernel, these are the raw resource management tasks such as memory management. While the higher level abstractions such as file systems, address spaces, and interprocess communication is done at the application level[1]. These abstractions are usually provided by library OSs which allow applications to handle their own machine resources in ways not possible with the traditional kernel. Which in turn can cause large performance boosts in several areas which will be shown below.

The exokernel walks this fine line between management and control by only providing three functions for accessing the machine's resources. It will only track ownership of resources, ensure protection by guarding all resource usage and bind points and revoke access to the resources. [1] By doing so the exokernel allows the library OSs maximum freedom over the machine's resources without allowing them to interfere with one another's resources as you would see in an unmanaged system.

Through these three functions the exokernel can control and allow many different situations. By tracking the ownership of the resources the exokernel can export privileged instructions to the library OS so that traditional OS abstractions can be implemented as well as allowing application based resource management which is the best way to build flexible and efficient systems all while avoiding resource management except when inter-library conflict protection is required to maintain system integrity.[1] By exposing the allocation and the raw resources and their physical names to the application layer the exokernel is able to allow the library to request physical resources which removes the expensive overhead that is involved with translating the virtual names to their physical names [3] also the physical names capture more useful information and are safer and less resource intensive. [3] Finally by exposing revocation the exokernel allows well behaved library OSs to preform application level resource management [1] and allows the library OS to choose what instance of the resource to release[1].

Not only does the exokernel benefit from it's decreased task load but the library OSs also experience several benefits compared to operating on a microkernel or VM. Library OSs running on an exokernel have a reduction in the number of kernel crossings compared to a microkernel[1] also since the library OS is not trusted by the exokernel it can then be trusted by the application. While a library OS may choose to handle low level management tasks it self, there is still a notion of portability for applications working with the library OSs such that applications that use a library OS that has implement standard interfaces, such as POSIX, will be portable on any system with the same interface [1] and a library OS can be made portable if it is designed to interact with a low-level machine independent level to hide hardware details. [1]

Exokernels follow the same design pattern of removing unnecessary code from within the kernel but without the same kernel to user space communication issues that microkernels experience. They also provide through their library OSs a simple yet effective way to emulate several different types of physical resource handling methods similar to virtual machines.

Paragraph 4 - Contrast/Compromise --[[User:Asoknack|Asoknack]]

Conclusion - Jon S. - Only a sentence per paragraph, excluding Intro

Sweet. Looks like we got it covered. We should read each others parts and put suggestions and edits. One of us should try and change it to one style if there are contradictions. And to put it on the main page. We can figure that out tomorrow. - Jon S

Once the other parts are up and you see anything you know of as a good reference to back it up, put the link so we can use it. -Slade

I made some edits to the first two paragraphs. I just reworded some of the unclear sentences and some grammatical errors. I'll work on editing more of it after comp 3007. Also when all the parts are up i can go through it and link the paragraphs together so it can be read more like an essay --[[User:Aellebla|Aellebla]] 15:18, 14 October 2010 (UTC)

So far so good, if you find some sentences that are off, go ahead and correct them, just note to us in here that you've made changes. Almost done guys! -Slade

Awesome Steph! Also, Awesome Corey, sounds sweet, looks good-JSlonosky

==Potential Test Questions==

Add potential test questions here:

Talk:COMP 3000 Essay 1 2010 Question 1

2010-10-14T22:37:24Z

Cling: /* The Essay */

== Microkernel ==
* Moving kernel functionality into processes contained in user space, e.g. file systems, drivers
* Keep basic functionality in kernel to handle sharing of resources
* Separation allows for manageability and security, corruption in one does not necessarily cause failure in system
* Large amount of moving from a process to Kernel to user space and back again, this is a costly operation.

----

''' Microkernel '''
* try's to minimize the amount of software that is mandatory or required [7]
advantages of Microkernel
* favors a modular system structure [7]
* one failure of a program does not impact any other programs [7]
* can support more than one api or strategies since all programs are separated [7]
==== Microkernel Concepts ====
* piece of code is allowed in the kernel only if moving it outside the kernel would adversely affect the system. [7]
* any subsystem program created must be independent of all other subsystem's, any subsystem that is used can guarantee this from all other subsystems [7]
===== Address Space =====
* a mapping that relates the physical page to the virtual page. [7]
* processor specific [7]
* hide's the hardware's concept of address space [7]
* based off the idea of recursion each subsystem has it's own address space [7]
* the micro kernel provides 3 operations [7]
** Grant [7]
*** allows the owner to give a page to a recipient, provided the recipient want's it the page is removed from the owner's address space and but in the recipients. [7]
*** must be available to the owner. [7]
** Map [7]
*** allows the user to share a page with a recipient [7]
*** page is not removed from the owner's address space. [7]
** Flush [7]
*** remove's the page from all recipients address space [7]
*** how does this work with Grant --[[User:Asoknack|Asoknack]] 19:10, 12 October 2010 (UTC)
* allows memory management and paging out side the kernel
* Map and flush is required for memory manger's and pagers [7]
* can be used to implement access right's [7]
* controlling I/O Right's and driver's are not done at kernel level [7]

===== Thread's IPC =====
* Threads
** in the kernel [7]
** Since a thread has an address space , all changes to the thread need to be done by the kernel [7]
* IPC [7]
** in the kernel IPC
** grant and map also need IPC (So buye the priciple above this has to be in the kernel)[7]
** basic way for sub process to communicate. [7]
* Interrupts
** partially in the kernel [7]
** hard ware is a set of thread's which are empty except for there unique sender id [7]
** transformation of the message to the interrupt is done in the kernel [7]
** the kernel is not involved in device - specific interrupt's and does not understand the interrupt. [7]
*** resting the interrupt is done at user level [7]
** if a privileged command is need it is done implicitly the next time an IPC command is sent from the device [7]

===== Unique Identifiers =====

== Virtual Machine ==
* Partitioning or virtualizing resources among OS virtualization running on top of host OS
* Virtualized OS believe running on full machine on its own

----
System Level Virtualization

=== VMM ===
* stands for Virtual Machine Monitor, also known as the hyper-visor[4]
* responsible for virtualization of hardware(mapping physical to virtual) and the VM that run on top of the virtuallized hardware [4]
* usually a small os with no drivers , so it is coupled with a linux distro that provides device / hardware access [4]
** the os that the VMM is using for driver's is called the hostOS [6]
*the hostOS provides login and physical access to the hardware as well as management for the VMM [6]
=== VM ===
* the OS that the vm is running is called the guestOS [6]
* the guestOS only sees resources that have been allocated to the VM [6]
==== three approaches ====
*Type I virtualization [5]
** runs off the physical hardware [4]
** Isolation of the guestOs from the hardware is done threw processe level protection meachnism[6]
*** ring 0 = VMM [6]
*** ring 1 = VM [6]
*** this means all instructions from the VM must go threw the VMM [6]
** since there can be multiple VM's on a computer the scheduling is done by the VMM [6]
** on boot the VMM creates a hardware platform for the VM [6]
** load's the VM kernel into virtual memory and then boot's it like a regular computer [6]
** ex. Xen [4]
*Type II virtualization [5]
** run off the host Os [4]
** ex. VMware , QEMU [4]
* Para-virtualization [6]
** Similar to Type but use the HostOs for Device driver access [6]
** Provide a virtualization that is similar to hardware [From the paper posted, no citation yet]
** GuestOS and Hypervisor work together to improve performance

----
==== ====
(Not complete but most of article 9)
Classical Virtualization
* VMMs allow programs in virtual environments to run natively other than resource usage
** Dominant instructions executed directly on cpu
** vmm completely controls system resources
** often need to emulate every native instruction which would severely effect the performance
** sensitive instruction that violate safety and encapsulation
** vmm handles them as priviledged instructions

x86 Virtualization
* virtualization in personal work stations rather than mainframes
** rings that allow isolation between virtual machines
** most privileged in ring 0 and least in ring 3. The operating system runs in ring 0 and user apps in ring 3
*** vmm in ring 0 and vms in lesser privilege rings (1 or 3)
*** guestOS believes its in ring 0
* address space compression, where to run the VMM
** if run using guest address space, guest can find out its virtualized or compromise the isolation
* does not trap all sensitive instructions but can handle them, violates classical virtualization description
* some privileged access fail without faulting
* interrupt virtualization - VMM handles AND guestOS handles
* binary translation - improve performance
* rewriting instructions and trapping before problems arrise

Paravirtualization
* guestOS become exposed to vm information so that the guest is aware that it is virtualized and can make decisions based on this
* allows to avoid problem instructions
* Xen
* guestOS must be modified and is not natively running
**works with the hostOS to run efficiently

VMM types
* hostedVMM - executes in hostOS and uses the drivers and support of the OS
* Stand-aloneVMM - runs directly on hardware and uses it's own drivers and services
* hybridVMM - runs a serviceOS where requests to hardware go through (I/O)

Device Emulation
* implement real hardware in software
* completely virtual device that the guest interacts with
* mapped to physical hardware that handles the interactions but the emulation allows conversion
* allows the vm to be easily migrated between machines as it does not rely on the physical hardware
* allows having multiple vms and simplifies sharing (multiplexing)
* poor performance as the vmm needs to do a lot to virtulize the machine

Paravirtualization
* modified guestOS to cooperate with VMM
* VMM does not have to do everything to handle device drivers
* not everything can be paravirtualized
* proprietary os and device drivers can't be paravirtualized
* still allows an increase in performance
* eventing or callback mechanism
** guestOS modifies interrupt mechs
* modifications are not applicable to all guestOS

Dedicated Devices
* does not virtualize device but assigns directly to guest vm
* uses guest's drivers instead of host
* simplifiest vmm by removing handing of i/o securily
* limited physical devices that can be dedicated
* dificult to migrate vm as it depends on the pairing with this resource
* elims over-head of virtualization and simplicity in vmm
* direct memory access not supported

== Exokernel ==
* Micro-kernel architecture with limited abstractions, ask for resource, get resource not resource abstraction
* Less functionality provided by kernel, security and handling of resource sharing
* Once application receives resource, it can use it as it wishes/in control
* Keep the basic kernel to handle allocating resources and sharing rather than developing straight to the hardware

----
* multiplex resources securely providing protection to mutual distrustful application threw the use of secure binding's[1]
* Goal of the exokernel is to give LibOS maximum freedom with out allowing them to interfere with each other. to do this the exokernel separates protection from management in doing this it provide 3 important tasks[1]
** tracking ownership of resources [1]
** ensuring protection by guarding all resource usage and binding points (not to shure what binding points are)[1]
** revoking access to the resources [1]
* LibrayOS (LibOs)
** Reduces the number of kernel crossings[1]
** Not trusted by the exokernel so can be trusted by the application , Example given is a bad parameter passed to the LibOs only the application is affected.[1] (So LibOs cant interact with kernel ???)
** Any application running on the Exokernel can change the LibrayOs freely [1]
** Application that use LibOS that implement standard interfaces (POSIX) will be portable on any system with the same interface [1]
** LibOs can be made portable if it is designed to interact with a low-level machine independent level to hide hardware details [1]

=== Exokernel Design ===
==== Design Principles ====
*Securely Expose Hardware [1]
** an Exokernel tries to create low level primitives that the hardware resources can be accessed from, this also includes interrupts,exceptions [1]
** the exokernel also export privileged instructions to the LibOS so that traditional OS abstractions can be implemented (eg Process , address pace)[1]
** Exokernels should avoid resource management except when required protection ( allocation , revocation , ownership)[1]
** application based resource management is the best way to build flexible efficient flexible systems [1]
*Expose allocation[1]
** allow LibOs to request physical resources [1]
** resource allocation should not be automatic, the LibOS should participate in every single allocation decision [1]
*Expose Names[1]
** Use physical name's when ever possible[3] (not to sure what physical names are, I think it is as simple as what the hardware is called)--[[User:Asoknack|Asoknack]] 20:27, 9 October 2010 (UTC)
** Physical names capture useful information [3]
*** safer than and less resource intensive than virtual names as no translations are needed[3]
*Expose Revocation [1]
** use visible revocation protocol [1]
** allows well behaved LibOS to preform application level resource management [1]
** Visible revocation allows the LibOS to choose what instance of the resource to release[1](Visible means that when revocation happens the exokernel tell the LibOS that resource is being revoked)
''' Policy '''
* LibOS handle resource policy decisions
* Exokernels have a policy to decided between competing LibOS (Priority , share of resources)
** it enforces this threw allocation and deallocation (every thing can achieved threw this even what block to write and such)

==== Secure Bindings ====
* Used by the exokernel to allow the LibOS to bind to resources [1]
* Allows the separation of protection and resource use [1]
* only checks authorization during bind time [1]
** Application's with complex needs for resources only authorized during bind.[1]
* access checking is done during access time and there is no need to understand complex resources needs during access[1]
** (this means that the exokernel checks once to make sure an application has authorization once approved, when the application tries to use the resource the exokernel is only concerned about policy conflict's)--[[User:Asoknack|Asoknack]] 18:20, 9 October 2010 (UTC)
** allows the kernel to protect the resources with out understanding what the resource is [1]
*three way's to implement
* Hardware Mechanisms [1]
* Software caching [1]
* Downloading application code [1]
''' Downloading Code to the Kernel '''
* used to implement secure bindings , and improve performance[1]
** eliminate the number of kernel crossings [1]
** downloaded code can be run with out the application to be scheduled [2]
==== Visible Resource Revocation ====
* Used for most resources [1]
** allows for LibOS to help with deallocation [1]
** LibOS are able to garner what resources are scare [1]
* Slower than Invisible as application involvement is required [1]
** ex of when invisible is used is Processor addressing-context identifiers [1]
==== Abort Protocol ====
* allows the exokernel to take resources away from the LibOS [1]
* used when the LibOS fails to respond to the revocation request [1]
* Exokernel must be careful not to delete as the LibOS might need to write some system critical data to the resource [1]

== Comparisons ==
====Exokernel/Microkernel====
'''Similarities'''
* Limited functionality in kernel
** functionality in kernel to handle sharing of resources and security
** avoids programming directly to hardware which creates a dependency
* Additional functionality provided in user space as processes
'''Differences'''
* Minimal abstractions provided by the kernel
** Applications given more power in exokernel

====Exokernel/VM====
'''Similarities'''
* Idea of partitioning resources between applications/OSs
* "Control" of resource given
* Isolation from other applications/OSs
'''Differences'''
* Exokernel runs applications, VM runs OS
* VM uses a hostOS and guestOSs run on top
* Virtualization on VMs, Exokernel deals with real resources
* VM hides a lot of information because it emulates. Exokernel does not.

====Microkernel/VM====
'''Differences'''
* With a virtual machine, you are not virtualizing apps like with a microkernel but virtualizing an entire Operating System.
* This can be costly but the benefits are that it's easier and all the standard OS features are available.

== References ==
[1]<nowiki> Engler, D. R., Kaashoek, M. F., and O'Toole, J. 1995. Exokernel: an operating system architecture for application-level resource management. In Proceedings of the Fifteenth ACM Symposium on Operating Systems Principles (Copper Mountain, Colorado, United States, December 03 - 06, 1995). M. B. Jones, Ed. SOSP '95. ACM, New York, NY, 251-266. DOI= http://doi.acm.org/10.1145/224056.224076 </nowiki>

[2]<nowiki>Engler, Dawson R. "The Exokernel Operating System Architecture." Diss. Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 1998. Web. 9 Oct. 2010. <http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.61.5054&rep=rep1&type=pdf>.</nowiki>

[3]<nowiki>Kaashoek, M. F., Engler, D. R., Ganger, G. R., Briceño, H. M., Hunt, R., Mazières, D., Pinckney, T., Grimm, R., Jannotti, J., and Mackenzie, K. 1997. Application performance and flexibility on exokernel systems. In Proceedings of the Sixteenth ACM Symposium on Operating Systems Principles (Saint Malo, France, October 05 - 08, 1997). W. M. Waite, Ed. SOSP '97. ACM, New York, NY, 52-65. DOI= http://doi.acm.org/10.1145/268998.266644 </nowiki>

[4]<nowiki>Vallee, G.; Naughton, T.; Engelmann, C.; Hong Ong; Scott, S.L.; , "System-Level Virtualization for High Performance Computing," Parallel, Distributed and Network-Based Processing, 2008. PDP 2008. 16th Euromicro Conference on , vol., no., pp.636-643, 13-15 Feb. 2008
DOI= http://doi.acm.org/10.1109/PDP.2008.85 </nowiki>

[5]<nowiki>Goldberg, R. P. 1973. Architecture of virtual machines. In Proceedings of the Workshop on Virtual Computer Systems (Cambridge, Massachusetts, United States, March 26 - 27, 1973). ACM, New York, NY, 74-112. DOI= http://doi.acm.org/10.1145/800122.803950 </nowiki>

[6]<nowiki>Vallee, G., Naughton, T., and Scott, S. L. 2007. System management software for virtual environments. In Proceedings of the 4th international Conference on Computing Frontiers (Ischia, Italy, May 07 - 09, 2007). CF '07. ACM, New York, NY, 153-160. DOI= http://doi.acm.org/10.1145/1242531.1242555 </nowiki>

[7]<nowiki>Liedtke, J. 1995. On micro-kernel construction. In Proceedings of the Fifteenth ACM Symposium on Operating Systems Principles (Copper Mountain, Colorado, United States, December 03 - 06, 1995). M. B. Jones, Ed. SOSP '95. ACM, New York, NY, 237-250. DOI= http://doi.acm.org/10.1145/224056.224075 </nowiki>

[8]<nowiki>Microkernel verses monolithic kernel
http://www.vmars.tuwien.ac.at/courses/akti12/journal/04ss/article_04ss_Roch.pdf - Roch</nowiki>

I will site it/reference it better later

[9]Fisher-Ogden J. 2006. Hardware Support for Efficient Virtualization. University of California, San Diego. http://cseweb.ucsd.edu/~jfisherogden/hardwareVirt.pdf

Not completely sure of the citation style used above.

== Unsorted ==
An overview of exokernels,virtual machines, microkernels *[http://www2.supchurch.org:10999/files/school/classes/CSCI4730/Lectures/grad-structures.ppt Overview](Power Point) 
Should not be used as a source but an overview.

The original paper on [http://portal.acm.org/citation.cfm?id=224076 Exokernels] --[[User:Gautam|Gautam]] 22:39, 6 October 2010 (UTC)

Exokernel-
Minimalistic abstractions for developers
Exokernels can be seen as a good compromise between virtual machines and microkernels in the sense that exokernels can give that low level access to developers similar to direct access through a protected layer and at the same time can contain enough hardware abstraction to allow similar benefit of hiding the hardware resources to application programs.
Exokernel – fewest hardware abstractions to developer
Microkernel - is the near-minimum amount of software that can provide the mechanisms needed to implement an operating system
Virtual machine is a simulation of any or devices requested by an application program
Exokenel – I’ve got a sound card
Virtual Machine – I’ve got the sound card you’re looking for, perfect virtual match
Microkernel – I’ve got sound card that plays Khazikstan sound format only
MicroKernel - Very small, very predictable, good for schedualing (QNX is a microkernel - POSIX compatable, benefits of running linux software like modern browsers)

This is some ideas I've got on this question, please contribute below
-Rovic

Outlining some main features here as I see them.

I found that the exokernel was an even lower-level design than the microkernel, closer to the hardware without abstraction. They have the same architecture with the basic functionality contained in the kernel to manage everyone. As the exokernel "gives" the resource to the application it can use the resource in isolation of other applications (until forced to shared) much like VMs receive their resources, either partitioned or virtualized, and execute as if its running on its own machine. There is this similar notion of partitioning the resources among applications/OS and allowing them to take control of what they have.

I'll locate some references later on. --[[User:Slay|Slay]] 15:00, 7 October 2010 (UTC)

I'm just going to post my answer for question 1 on the individuel assignment and hope it helps. --[[User:Aellebla|Aellebla]] 15:06, 12 October 2010 (UTC)

The design of the micro kernel was to take everything they could out of the Kernel and put it into a process. For ex, networking would be put into a process instead of staying in the kernel. The micro kernel dev's tried to keep lots of things in user space for efficiency. But one major problem with this is there would be a large amount of moving from a process to the kernel to user space and back again and this is a costly, non efficient process.It was an application specific OS, there was no multiplexing. With a virtual machine you are not virtualizing apps like with a microkernel but virtualizing an entire Operating System. This is very heavy however but the benefits are that it‟s easy and all the standard OS features are there whereas in a microkernel setup they would not all be there and this can be seen as a compromise.

Exokernels can be seen as a compromise to virtual machines and microkernels because virtual machines emulate and exokernels do not. When you emulate something you hide a lot of the actual information because you wouldn‟t be able to see the „real‟ hardware. If we look at a virtual box setup running Linux, and we go look at all the hardware, it will be displayed as fake hardware.

Maybe we can have an introduction - paragraph or so on each type - then similarities - differences - and the compromise. I am going to do some research and writing this weekend and I will put some up -- Jslonosky

btw in my page (i guess you can call it that) i have some resources i have found --[[User:Asoknack|Asoknack]] 15:50, 8 October 2010 (UTC)
- Wow, nice man. I will go ahead and write up the descriptive paragraphs on each kernel and virtual machine if no one minds. --Jslonosky

I think we should divide up the paragraphs and proofread each others instead. (Are there only 4 of us?) I don't have much time to work on this today though but I'll try to work on it tomorrow morning. - Slay

Sure guy. That sounds good. There should be 5 or 6 of us though.. . Oh well. Their loss. I will do some before or after work today. Ill start with Microkernel since there is not a large amount of info here, and so we don't overlap each other - JSlonosky

yeah i think there was more like 7 of us btw if any one has any more information feel free to add it would be nice if you add the references so that way citing is really easy on acm.org it will auto give you the citation info (where it says Display Formats click on ACM Ref and new window with the citation info auto pop's up) --[[User:Asoknack|Asoknack]] 02:28, 11 October 2010 (UTC)

I added an outline of the similarities and differences. Add any more that I missed. These are from observations so I don't have any resources. -Slay
That's probably fine. Our textbook probably outlines some of them, so I am sure we can find a few there - JSlonosky

Talked to the teacher today and for VM he said we should focus on the implementation such as Xen and VMware , he also said to talk about para virtualization --[[User:Asoknack|Asoknack]] 18:42, 12 October 2010 (UTC)

A paper about emulation and paravirtualization [http://portal.acm.org/citation.cfm?id=1189289&coll=GUIDE&dl=GUIDE&CFID=105648137&CFTOKEN=47153176&ret=1#Fulltext link] - Slay

Oh no big words. Sorry about the Microkernels not done yet. Working on an outline now. Finally found how to access the ACM through carleton. Gawd.
I am planning an outline, quick bit about kernels in general, (maybe mention monolith kernels?), and what microkernels do.
I see the microkernel outline info and a reference ( Whomever did that == hero: true) about the scheduling and the Memory management. Should that be included in kernels in general and then mention what microkernels build upon/change? - JSlonosky

Sorry late to the party here. My mistake was not checking the discussion page when I checked in. I don't want to trample anyone's current work but I don't see any work on the final essay done. I would love to help just need to know where I can step in so as to not screw anyone else up. -- [[User:Cling|Cling]]

I don't think I'll be able to write up something for the final essay, even though I suggested splitting it. I'll do research tonight though on the paravirtualization. If I find the time, I'll try to write something. Sorry about that. --[[User:Slay|Slay]] 21:52, 13 October 2010 (UTC)

We all have 3004 to do too, man. I do not think anyone has chosen to do Virtual Machine section yet, or the Exokernel itself. But the contrast paragraph and the intro is chosen, and intro is done. Microkernel and kernel will be done in a hour I hope. -- JSlonosky

I can attempt to write up anything, the issue is I don't have any context on what to write, how do I tie it in to the rest of the essay? I only have a Japanese Quiz tomorrow morning then I should be good to write anything up for the rest of the day. As someone who has already written part of the essay, and assuming I attempt the exokernel section, how much do you think I should write? Should it just be about exokernel or should there be comparisons to the other topics? Thanks --[[User:Cling|Cling]] 23:14, 13 October 2010 (UTC)

Go with the Exokernel itself. Slade is getting off work in a hour and we can double check what he is doing then. We can put it together tomorrow sometime, and fill in the other stuff. - JSLonosky

I'll attempt to work on VM tonight, then. I would feel so bad if I didn't write anything. -Slay

Still wondering how much to write, I think we should decide on a decent word count or length so we don't have one short section (which would probably be mine) and/or one massive section that dwarfs all the others. If anyone has already written a section could you post your word count so we can aim to be around there, it would obviously be just a recommendation but it's just better to be on the safe side and have everything uniform. I haven't seen any formal requirements for the essay but I could be wrong, I also haven't been to class in a while. --[[User:Cling|Cling]] 23:33, 13 October 2010 (UTC)

Yeah Slay, VM probably doesnt have much to write about. Get something down, and we can go over it. CLing, Just write what you think. There is not a lot to go over if I write kernel/microkernel well enough. What is a exokernel? exokernel was an even lower-level design than the microkernel, closer to the hardware without abstraction, basically (As said by Slade). I will probably end up with 500 or a bit more words. -- JSlonosky

Sound off!

Who's actually reading this? Add your name to the list...

Rovic P.
Jon Slonosky
Corey Ling
Steph Lay
Aaron .L

== The Essay ==

Let's actually breakdown the essay into components then write it here.

I'd like to go along the premise that microkernels and and virtual machines are "weaker" than exokernels in design for the essay. If anyone has any objections, add it here.

-Slade

what do you mean by "weaker"(i think you mean exokernels' takes the best of both worlds ) --[[User:Asoknack|Asoknack]] 02:45, 13 October 2010 (UTC)

What I mean by weaker is that we should focus on the things microkernels and virtual machines may not do as well compared to a system based off an exokernel design and then focus on how an exokenenel can take the best of both worlds. Please choose which section you will work on, that's not to say it'll be the only part you do, but rather we'll all contribute to each part please. 1 day left.
-Slade

...to the extent that exokernels be seen as a compromise between virtual machines and microkernels.
-I'll work on the initial intro. -Slade

3 paragraphs that prove it
Explain how the key design characteristics of these three system architectures compare with each other.

intro/thesis statement -Rovic P.

In Computer Science, the kernel is the component at the center of the majority of operating systems. The kernel is a bridge for applications to access the hardware level. It is responsible for managing the system's resources such as memory, disk storage, task management and networking. We are comparing Exokernels to Microkernels and Virtual Machines by looking at how the kernel goes about such management and its connections. In the Exokernel conceptual model, we can see exokernels become much smaller than microkernels because as this design shows, they are tiny and strive to keep functionality limited to protection and multiplexing of resources. The Virtual Machine Implementation of virtualizing all devices on the system may provide compatibility, but it also adds a layer of complexity within the system. This is less efficient than a real machine as it accesses the hardware indirectly. It can be observed by examining how the exokernel provides low level hardware access and provides custom abstraction to those devices. This is done in order to improve program performance as opposed to a VM's implementation. The exokernel concept has a design that can take the better concepts of microkernels and virtual machines to the extent that exokernels can be seen as a compromise between a virtual machine and a microkernel.

Paragraph 1 -Microkernel -Jon S.

The kernel is the most important part of an operating system. An operating system could not function without the kernel.

A kernel is the lowest level section of an operating system. Within a system, it has the most privileges. It runs along side of the ‘user space’. It is in the ‘user space’ where a user has access. This is also where the user can run its applications and libraries.[8] This leaves the kernel with the need to manage the other necessary processes. For example, the kernel could manage the File Systems and complete process scheduling. The kernel is layered with the most authoritative process on its lowest level.[8] A monolithic kernel, which is a kernel that contains all mandatory processes within itself, was the common kernel type of the earlier versions of today’s operating systems utilized. However, this architecture had problems. [8] If the kernel needed to be updated with more code, or a change in the system, the entire kernel would need to be compiled. Therefore, due to the amount of processes within it, it would take an inefficient amount of time. Here, a microkernel becomes practical.

The concept of a microkernel, is to reduce the code within the kernel. The microkernel is only included in the kernel if it would impact the system. There are a variety of ways the system could be affected if a microkernel were to be implemented, for example, there would be increased performance and efficiency. [7] Furthermore, a microkernel is a kernel that has a reduced amount of mandatory software within itself. This means that it contains less software to manage, and has a reduced size.

A microkernel that emerged at the end of the 1980’s to the early 1990’s has the structure that processes as if the File Systems and the Drivers are removed from it, leaving the kernel with process control and input/out control, and interrupts. [8] This new structure makes the system much more modular, and easier to provide solutions. If a driver must be patched or upgraded, the kernel does not need to be recompiled. [7] The old driver can be removed, and during the time the device waits for the system to recognize it, the operating system replaces the driver. This allows for real-time updating, that can be done while the computer is still functional. This can reduce the complete crash of the system. Therefore,if a device fails, the kernel will not crash itself, like a monolithic kernel would. The microkernel can reload the driver of the device that failed and continue functioning. [7]

Want more on the scheduling? I can do that if wanted. -key note on exokernel's mutiplexing vs microkernel's messaging, exo more efficient so perhaps running with the idea that messaging b/w processes not necessarily the ideal way need to also start outlaying weaknesses in the design as well in order to play up the idea that an exokernel just does it better -Slade

Paragraph 2 -Virtual Machine -Steph L.

A Virtual Machine, or VM, is a software abstraction of a physical machine. This entails virtualization of the physical machines resources in order to share them among OS run in the VM. Virtualizing these resources allow the OS to run as if it were on a full machine when, in reality, it is actually running in a virtualized environment on top of a hostOS. The OS is actually running on the machine, sharing the resources.

Virtual Machines generally contain two key components; both the Virtual Machine Monitor, or VMM and the VM.

The VMM, also known as the hypervisor, manages the virtualization of the physical resources and the interactions with the VM running on top. [4] In other words, it mediates between the virtualized world and the physical world, keeping them separate and monitoring their interactions with each other. The hypervisor is what allows the VM to operate as if it were on its' own machine. This is done by handling any requests to resources and maintaining these requests with what has actually been provided to the VM, by the hostOS. The hostOS provides management for the VMM, as well as, allowing physical access to devices, hardware and drivers. [6]

The VM, is what contains the OS, we are running through virtualization. [6] This OS is called the guestOS. It will only be able to access any resources that have been made available to the VM by the hostOS. [6] Otherwise, the guestOS will not know about any other resources and does not have direct access to physical hardware. This will be taken care of by the VMM. While, the guestOS will execute as its own machine, unaware of this mediator.

There are various ways of implementing hardware virtualization in a system to allow VMs to run. This includes device emulation, paravirtualization and dedicated devices. [9]

In device emulation the VMM provides a complete virtualization of a device for the guestOS to interact with, in the software. [9] The VMM will map this virtualized device to the physical resource and handle any interactions between them. This will usually include converting instructions from the guestOS into instructions that are compatible with the device. [9] Device emulation allows for the VM to be migrated easily to another machine as it is not dependent on the physical devices but rather, on the software emulations instead. [9] It also allows for simpler multiplexing between multiple virtual machines as it can handle sharing though these virtualized devices. [9] A drawback of emulation, however, is poor performance. This is because the VMM must handle every request and convert them to be compatible with the physical device. [9] Nonetheless, despite its poor performance, emulation is still the most common form of virtualization.

Paravirtualization allows for a boost in performance by having the guestOS and the hostOS work together to improve performance. [9] In paravirtualization, the guestOS is not a native OS and must be modified so guestOS is aware that it is a virtualized system. [9] Since the guestOS is aware of this, it can now make better decisions about how it accesses devices. Seeing as the guestOS will be able to handle its decisions better, the VMM’s responsibility is reduced as it now does not have to translate between the guestOS and the physical devices. [9] Though the performance boost is a great advantage, there are many disadvantages to this. Some disadvantages are that you can only use paravirtualization if you can implement the modifications to the guestOS. As well, not everything can be paravirtualized and as such, this limits the cases in which this method can be used. [9] Also, every guestOS must be modified in order to be used in paravirtualization. The modifications will differ in various OS and so, there is also the task of implementing these changes to make a guestOS compatible. [9]

Instead of virtualizing the hardware and mediating between the VMM, dedicated devices allow mapping directly to the guestOS. [9] In this method, the device will use the guestOS’s drivers instead of the hostOS’s. [9] Using this method allows the guestOS to use the hardware to its full extent, without having to deal with the VMM. This then, simplifies the VMM by eliminating the overhead by virtualizating the hardware and handling the requests to devices. [9] However, there are limited physical resources to be dedicated to a guestOS. This also makes migration difficult as the guestOS is dependent on the physical device. [9]

'''notes'''
- it ended up being quite lengthy. I mainly focused on the device virtualization rather than the architecture of a VM (like x86 virtualization). I'll put up my notes for the paper I found for virtualization. I didn't talk about Xen or VMware though. If any of that is needed, I can try to continue working on it tonight but I have another priority.

-try focusing on the emulation side of VM where emulation's weaknesses vs direct hardware access or custom abstraction that exokernels -Slade

Paragraph 3 -Exokernel -Corey L

(This is only half, I have a Bell Tech here working on my internet. I will finish when he leaves)

Exokernels can be seen as a compromise between a microkernel and a VM. It can also be seen as simply dividing a monolithic kernel up into 2 parts. The management tasks of the kernel remain in the exokernel, these are the raw resource management tasks such as memory management. While the higher level abstractions such as file systems, address spaces, and interprocess communication is done at the application level[1]. These abstractions are usually provided by library OSs which allow applications to handle their own machine resources in ways not possible with the traditional kernel. Which in turn can cause large performance boosts in several areas which will be shown below.

The exokernel walks this fine line between management and control by only providing three functions for accessing the machine's resources. It will only track ownership of resources, ensure protection by guarding all resource usage and bind points and revoke access to the resources. [1] By doing so the exokernel allows the library OSs maximum freedom over the machine's resources without allowing them to interfere with one another's resources as you would see in an unmanaged system.

Through these three functions the exokernel can control and allow many different situations. By tracking the ownership of the resources the exokernel can export privileged instructions to the library OS so that traditional OS abstractions can be implemented as well as allowing application based resource management which is the best way to build flexible and efficient systems all while avoiding resource management except when inter-library conflict protection is required to maintain system integrity.[1] By exposing the allocation and the raw resources and their physical names to the application layer the exokernel is able to allow LibOs to request physical resources

Paragraph 4 - Contrast/Compromise --[[User:Asoknack|Asoknack]]

Conclusion - Jon S. - Only a sentence per paragraph, excluding Intro

Sweet. Looks like we got it covered. We should read each others parts and put suggestions and edits. One of us should try and change it to one style if there are contradictions. And to put it on the main page. We can figure that out tomorrow. - Jon S

Once the other parts are up and you see anything you know of as a good reference to back it up, put the link so we can use it. -Slade

I made some edits to the first two paragraphs. I just reworded some of the unclear sentences and some grammatical errors. I'll work on editing more of it after comp 3007. Also when all the parts are up i can go through it and link the paragraphs together so it can be read more like an essay --[[User:Aellebla|Aellebla]] 15:18, 14 October 2010 (UTC)

So far so good, if you find some sentences that are off, go ahead and correct them, just note to us in here that you've made changes. Almost done guys! -Slade

Awesome Steph! -JSlonosky

==Potential Test Questions==

Add potential test questions here:

Talk:COMP 3000 Essay 1 2010 Question 1

2010-10-14T00:35:21Z

Cling: /* Unsorted */

Talk:COMP 3000 Essay 1 2010 Question 1

2010-10-14T00:21:12Z

Cling: /* The Essay */

Talk:COMP 3000 Essay 1 2010 Question 1

2010-10-13T23:33:13Z

Cling: /* Unsorted */

Talk:COMP 3000 Essay 1 2010 Question 1

2010-10-13T23:14:01Z

Cling: /* Unsorted */

Talk:COMP 3000 Essay 1 2010 Question 1

2010-10-13T21:12:13Z

Cling: Introduction