Soma-notes - User contributions [en]

Talk:COMP 3000 Essay 2 2010 Question 8

2010-12-02T04:34:46Z

Gbint: /* Work In Progress */

Group Members

Trevor Bonesaw Malone - tmalone@connect.carleton.ca //FIRST POST!

Qi Zhang - qzhang13@connect.carleton.ca

Gregory Bint - gbint@connect.carleton.ca

Gautam Akiwate - gakiwate@connect.carleton.ca

Corey Ling - cling@connect.carleton.ca

Sarah Liske

== Work Plan ==

As Trevor intimated, we should have clear division of work going forward. This is sort of the break down as I see it. Please edit as you think of new ideas!

* Background Concepts
** Information Flow Theory. (Implicit and Explicit Flows.) --Done[--[[User:Gautam|Gautam]] 03:54, 28 November 2010 (UTC)]
** What is dynamic taint analysis --Done[--[[User:Gautam|Gautam]] 05:07, 28 November 2010 (UTC)]
** What is the difference between dynamic and static analysis --Done[--[[User:Gautam|Gautam]] 03:54, 30 November 2010 (UTC)]]
* Research Problem
** How do we build a DTA engine for a phone? - done, but by who?
** Why do we want to? (information misuse) - done, but by who?
* Contribution
** How did they implement their DTA engine (Done: --[[User:Cling|Cling]] 04:50, 26 November 2010 (UTC))
** What did they find about information misuse (Done: --[[User:Cling|Cling]] 04:50, 26 November 2010 (UTC))
** Compared to the existing taint tracking approaches. [[User:Zhangqi|Zhangqi]] 07:11, 27 November 2010 (UTC)
** (What else should be in the contributions? Anything need fleshing out?) (Working on that now :) ) sliske
* Critique
**Added two paragraphs at the end of the present critique. Please incorporate it into your content as you deem fit.--[[User:Gautam|Gautam]] 09:07, 30 November 2010 (UTC)
**^ done. fleshed out critique, and added a bit about how taintdroid doesn't track implicit flow. Also reworded (the entire essay) for clarity where necessary/checked spelling. It would be a good idea for everyone to read it over once for spelling/clarity before thursday, just in case something doesn't make sense - sliske
* References
** The article has 61 references! We can probably use some of them
**whee! reading papers and sticking in information as need be. Also working out how to cite properly, as there are two citations used currently
**references added and citations taken care of. will go over fill in a few places where information may be lacking after class sliske
**Referencing is a little askew. The numbers don't match the papers as listed in the referencing. Also the papers are usually cited with a number and enclosed in "[]"
**thanks for giving the paper a read over/noticing that :)

List of information we need to find external sources for:
* History of taint analysis
* History of privacy research relating to smart phones

== Work In Progress ==

Log what you are working on *right now* so that other people don't try to do the same thing. Make sure to clear your name from here when you are done.

* Gregory Bint: Research Problem

* Gautam Akiwate: Background Concepts
** Any resources on Dynamic taint Analysis would be appreciated!

* Qi Zhang, Corey Ling: Contributions

* Trevor Malone: Critique

* Sarah Liske: References and Questions, Clarity/Spelling.

== Some Notes from the Video ==

Tracking of privacy sensitive data through Dynamic Taint Analysis (aka. Taint Tracking). The trick is to mark private data as it sourced, and then follow those marks until (unless) they leave the phone.

Android phones run Java apps, which are compiled into DEX, and then run on top of the Dalvik VM. It is this VM that we modify so that we can support the storage and tracking of taint tags.

Taint sources
* low -bandwidth sensors
** Location
** Accelerometer
* High-bandwidth sensors
** Mic
** Camera
* Information DB
** Address book
** SMS storage
* Device ID
** IMEI
** IMSI (don't actually track this one because of false positives)
** ICC_ID
** Phone Number

Taint sink (where marked data can leave the phone)
* Network Taint Sink

Taint propagation
* ???

Taint tags are stored in memory interleaved with the variables they are tracking

Some standard Data Flow technique is used to propagate these tags, especially as one variable that is marked may be assigned to another, so now that variable needs to be tracked as well.

Tracks explicit flows of data, not implicit
To fully capture implicit flows, you need to do static analysis, which is hard with closed-source apps, and cannot be done real-time

Implicit flows are not tracked
* Implicit flows can involve "taint-scope", tracking based on conditionals in code

=== Performance ===

The goal is to create a real time tracking system, so the TaintDroid's performance impact is of some importance

14% CPU overhead
4.4% memory overhead

Macro benchmarks (to get a feel for what the phone's usability is like with TD running)
* App load: 3% (2ms)

=== Findings ===

20 out of 30 tested applications share data in a way that is not expected.

67 of 105 flagged pieces of data leaving the device had no obviously legitimate purpose (verified by the authors).

Many apps sent location data and other unique identifiers to advertising servers.

Most apps do not mention anything to the user.

=== Limitations ===

Tracks only explicit data flows.

An application *could* launder the tags off of the data, if they really wanted to hide this sort of thing from TaintDroid.

There are methods that could be used to protect against this, but they go against the goal of a light-weight, real-time tracking system. TD is not necessarily about catching truly malicious programs, but rather just those that leak information.

Why do apps take this information?
* Lazy; in the demo video, the wallpaper app seems to use the IMEI just as a ready made unique ID
* Overzealous; the developer might thing they *need* the data for something, but actually
* Ads; advertises do seem a little presumptuous in their data collection
* Spying; bosses or spouses
* Malicious;

=== QA Period ===

Q: how do we prevent a malicious app from removing a taint attribute on a file

A: TD operates a too low a level for this to be a problem; TD assumes that the native code is trusted

Q: It seems like you had a lot of false positives

A: The point of this tool was to identify privacy sensitive information as having left the phone, not whether or not a privacy violation has taken place.

Q: Now that TD is released; couldn't malicious apps use some of the methods described in the paper to get around it?

A: Well, yes, but it is not just about maliciousness, it could just laziness or over-zealous ad stuff.

==Other Information==

Hey guys, thought I would just post a generalized paragraph about our essay.

In today’s society, Smartphones are the new big thing. To me that’s what makes this paper so interesting. This paper focuses on private information in android phones and the misuse of this information. The misuse of information includes the SIM card, the ID of the device, or the phone number. TaintDroid is used on smart phones with an efficient taint tracking and analysis system. It has the ability to track sensitive data from multiple sources and examines the misuse of such data. In their study, out of 80 popular third-party applications, TaintDroid monitored that 68 applications had potential misuse of user’s private data. This tool is great for knowing with applications are safe and which are not, so your private data can remained private.

Also, we should really think of splitting up the work in some way. If some people have specific sections they would like to do lets figure that out now so we can divide the workload and get it done over the next couple of days. I don't personally care what part I'm going to have to do, so lets get this going. Any other information people wanna post feel free the more the better, even if we don't end up using it.

[[user:Tmalone|Trevor Malone]]

Hey guys! Anything else we need to get done? Let me know and I can help in anyway possible.

[[user:Tmalone|Trevor Malone]]

==Relevant Sources==
*NEWSOME,J.,AND SONG,D.Dynamic Taint Analysis for Automatic Detection, Analysis, and Signature Generation of Exploits on Commodity Software. [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.83.2141&rep=rep1&type=pdf Dynamic Taint Analysis for Automatic Detection]
Seems to be THE Dynamic Taint Analysis Paper.Talks about implementation on TaintCheck. Could be also useful for critique section -[Gautam]

COMP 3000 Essay 2 2010 Question 8

2010-11-30T22:40:22Z

Gbint: /* Questions */ answers

=TaintDroid: An Information-Flow Tracking System for Realtime Privacy Monitoring on Smartphones=
'''Authors:''' 
* William Enck, Patrick McDaniel ''The Pennsylvania State University'' 
* Peter Gilbert, Landon P. Cox ''Duke University'' 
* Byung-Gon Chun, Jaeyeon Jung Anmol N. Sheth ''Intel Labs''

[http://appanalysis.org/tdroid10.pdf Direct Link]

[http://www.appanalysis.org/ Official Website]

[http://www.youtube.com/watch?v=qnLujX1Dw4Y Video Demonstration]

=Background Concepts=
To follow these ideas in this paper, the ideas which form the basis of this theory have to be understood. All in all, the following two concepts can be said to be central to understanding this paper. 
==Information Flow==
Information flow as the name suggests is the transfer of information. This transfer of information can be between two processes or within a given process, for example, between variables. [REF] Information Flow Theory tries to quantify this flow of information into a mathematical model. 
In a security model the information flow can be categorized into: 
===Explicit Flow===
Explicit flow is when information subject to security classifications is transferred to a variable or process which is not subject to the same or higher level of security, causing a security breach. [REF] The breach occurs because information is now more visible than it was intended to be. An example of explicit flow is shown below:
 
<code>PRIVATE VAR <big>secure</big> 
PUBLIC VAR <big>notsecure 
notsecure = secure</big> </code>
 
The information in 'secure' which is PRIVATE is transferred to 'notsecure' which is PUBLIC which is an information leak.
 

===Implicit Flow===
Implicit Flow is when information subject to security classifications is deduced indirectly. This the leakage of information occurs through the program control flow. [REF] Depending on the flow of the program the secure information can be compromised, as shown below 
 
<code>PRIVATE VAR <big>secure</big> 
PUBLIC VAR <big>notsecure 
if secure="blah blah" then: 
insecure=1 
else: 
insecure=0</big> </code>
 
Since can determine the value of information in secure using logic statements, we can indirectly access the secure information. Information leak due to implicit flows is much harder to detect and protect from, due to the indirect nature of implicit flows.

==Taint Analysis==

The basic premise of taint analysis is to follow the information flow of "tainted" variables to ensure that they do not create a security breach. Any variable that can be modified directly or indirectly by the user and can become a security vulnerability is "tainted". Through various operations the "taint" can be passed from variable to variable, propagating it. When a tainted variable is used to execute potentially dangerous commands a breach is logged, allowing detection of possible security concerns. 

===Dynamic Taint Analysis===
Taint Analysis done at run-time is called as Dynamic Taint Analysis. The approach used in dynamic taint analysis is to label data originating from untrusted sources as tainted. The analysis keeps track of all the tainted data in memory and when such data is used in a potentially dangerous situation, a leak is detected. This approach offers the capabilities to detect most input validation vulnerabilities with a very low false positive rate. However, the execution of the program is slower because of the additional checks being preformed. 

===Static Taint Analysis===
Static taint analysis is the technique used for detecting the over approximation of the set of instructions that are influenced by user input. The set of tainted instructions is computed statically by analyzing the sources of the program. The main advantage for static taint analysis is that it takes into account all the possible execution paths of the program. On the other hand the analysis may not be as accurate as a dynamic analysis because the static analysis does not have access to any additional run-time information of the program. 

===Mathematical Model===
<code>For all variables V = {T,U} ;T are tainted and U are untainted: 
Using <big>⊕</big>: V x V -> V, x, y ∈ V 
<big>
x<big>⊕</big>y = T; x = T OR y = T 
x<big>⊕</big>y = U, if x = U AND y = U
</big></code> 

It is now easy to see that whenever a tainted variable is used by another variable, the variable that used the tainted variable becomes tainted as well; the taint is propagated. Taking this further we can see that, if needed, we can tag variables as tainted by attaching to them a tainted tag, which can then be tracked or used as wanted. 

Note: The paper talks about Dynamic Taint Analysis. TaintDroid makes ingenious use of "taint" to taint variables that are of value and tracks their progress. Though in the actual context of Taint Analysis "taint" is used for untrusted information however in this case the "taint" variables are infact important private data. (Just in case if it confused someone :D) For more detailed information on Taint Analysis refer "Detecting Software Vulnerabilities Static Taint Analysis"[2]

Explain briefly the background concepts and ideas that your fellow classmates will need to know first in order to understand your assigned paper.
* Background on the taint data tracking method, how it has been used in other systems (i.e. not phones)
* A reader's digest version of any new articles about this kind of security vulnerability on phones, on apps that collect more personal data than users would expect.

=Research problem=

In today’s society, smartphones are a prominent new technology. Smartphones, by their nature, are linked into many private details of our lives, including not only classic data like our contact list, but new kinds of data smartphones make available, such as location data. Smartphones also have the ability to download and run third party applications which can connect to the internet; indeed, this is why we call them "smart". Except for the odd tunnel or elevator, these phones are constantly connected to the internet. When you combine third party applications with an internet connection on a device that stores an immense amount of personal data, you suddenly find yourself unsure of how your data is being used; what is to stop a third party application from disseminating our private information? As it turns out, very little.

A telling example of this is a wallpaper application that sends your phone number back to the developer. Once the app is running on your phone, it can typically access any of the information on your phone that it has been given permission to access, and it is not necessarily clear when the application has accessed data, or what it is doing with it.

The authors of this paper set out to try to understand what kind of information is being collected and where that information is being sent, and in order to do that, they first needed to build a means of tracking that information.

The strategy they chose is called Dynamic Taint Analysis, sometimes called Taint Tracking. The basic idea being to mark (or ''taint'') sensitive information at its source, and to then follow that mark as the data moves through a system. In the context of this paper, if ever we should see marked data leave the network interface of the phone, then we know that some sensitive information has been disseminated.

There are many difficulties associated with implementing such a system on a smartphone. Their design goals were to create a light-weight, minimal overhead, real-time tracking system that runs directly on a real phone, with real applications. To be really useful, the tracking system must not impact the user experience too heavily.

Some implementation difficulties are:
* Smart phones are resource constrained. Processing power and memory are limited, and any processing that we do perform will consume battery power. If the tracking system is to be real-time, the phone must be considered "usable" by the end user, and so the system must be truly light weight.
* Third party applications arrive in a compiled format; we cannot analyze their source code.
* Applications may do complex things with the sensitive data. It is unlikely that the application will simply read a location from the GPS and dump it straight out over the network. More likely is that the application will use that data in some way, or combine it with other data, before it is sent. We need to be able to track sensitive data throughout this entire process if we hope to perform any useful analysis.
* Applications can share information with other applications, meaning that our tracking has to work across multiple processes.
* The tracking must operate on a real phone, not a simulated one. With a simulated system, where we control the virtual hardware and memory, we can be certain that we can see everything that an application might do. On a real device, how can we get low-level enough to see everything the applications do?

How does this problem relate to past related work?

=Contribution=
The main contribution of the TaintDroid paper is not that they achieved information flow tracking, but that they made it efficient enough to run in real time on real constrained hardware devices with minimal overhead. As stated "TaintDroid only incurs an approximate 14% CPU overhead and an approximate 4.4% memory overhead for simultaneously tracking 32 taint markings per data unit." [CITE] It should also be noted that the 14% CPU over-head is only in regards to a "CPU-bound micro-benchmark and imposes negligible overhead on interactive third-party applications."

This low overhead is achieved by modifying the code directly at the Java Virtual Machine (JVM) layer of the Android system to provide variable-level tracking. This allows direct control over how and what private information, such as location details from the GPS, is stored and accessed. Next, they modified the Java Native Interface (JNI) to provide message-level tracking which allows them to monitor inter-process, a.k.a. inter-application, communications. This also allows them to "patch the taint propagation on return." [CITE] so they can keep track of information transfer via native code. Finally, by modifying the network interface and secondary storage interfaces they are able to provide file-level taint tracking which enables them to ensure "persistent information conservatively retains its taint markings." [CITE].

Another contribution of TaintDroid is accuracy of tracking sensitive data. Unlike existing solutions that rely on heavy-weight, whole-system emulation, the virtualized architecture of Android allows four levels of taint propagation: variable, method, message, and file. The granularity and flow semantics that TaintDroid offers highly influences the performance and accuracy of TaintDroid. Existing taint tracking approaches, like Panorama Taint System, rely on instruction-level dynamic taint analysis using whole system emulation. This method can lead to the system preforming from 2-20 times slower than normal, which is not suitable at all for the trend of realtime analysis. Moreover, instruction-level tracking faces a serious problem, taint explosion. When we use some complex instructions such as CMPXCHG, REP MOV,the stack pointer may become falsely tainted or taint loss. However, TaintDorid solved this problem with the combination of 4 levels of tracking. For example, the variable level allows TaintDroid to provide flow semantics for taint propagation, allowing distinction between different data pointers at different levels to ensure accuracy.

By combining these four levels (variable, method, message and file) of taint tracking, TaintDroid was able to effectively track 30 randomly selected, popular, 3rd party android applications. In doing so TaintDroid correctly flagged 105 instances of tainted information transmission. Of these 105, only 35 were legitimate risks. It also determined that 50% of the applications submitted the users location to advertising servers and 5 of the applications transmitted the users device ID, phone number and SIM card serial number. Clearly, the higher granularity is needed and TaintDroid is providing a step in the right direction, by providing a highly efficient real time tracking system.

=Critique=

==Style==
This paper has quite a bit of information, but has a very strong structure in explaining what TaintDroid is and what it does, which makes it easy to read. The paper begins with a high-level overview of TaintDroid, then explains the history followed by an explanation of sources that are tracked by TaintDroid and its design. It continues with test results and the strengths and weaknesses of TaintDroid, with references to related work. 
Readability of the paper could be improved by explaining more of the Android phone architecture first, before beginning the overview of the TaintDroid process. That way the concepts of the implementation would be easier to understand. Another possible improvement would be for this paper to provide some solutions to the problem it found, by suggesting future smartphone security measures, or what possible updates to current programs can be implemented to further improve the tracking of misuse of information, and even prevent it from occurring.

==Content==
Challenges of monitoring network disclosure of privacy sensitive information are well outlined, as are TaintDroid's workarounds for these challenges. TaintDroid uses dynamic taint analysis to find a way around the challenges, using a taint source as the targeted sensitive information, and a taint marking to identify the information type. It is easy to see that this research was effective, due to the impressive number of information leaks that were found. TaintDroid effectively identifies information misuse at a high percentage. However, while the implementation is strong in that the overhead is so low and accuracy is high, there are trade-offs that were incurred to meet that overhead. For example, TaintDroid only tracks data flow and does not track control flow, which means there are ways to bypass the TaintDroid filters. There are also other issues, particularly in Taint Tag Storage, which are due to the fact that most string objects have the same tag. Because of the similar tags, it is possible for false positives to occur. 
Further more, TaintDroid is a firmware modification, not an application which raises the questions of its usability by the average user. Being a firmware modification drastically reduces its usability unless 'Android System' itself incorporates these changes which is highly unlikely as the overheads, in this case a memory overhead of 4.4%, an IPC overhead of 27% and an overall 14% overhead, are on the higher side in an already resource constrained smartphone. 
For example, consider a possible alternative implementation of TaintDroid. TaintDroid is incorporated in the firmware and hence incurs an additional overhead as the user uses the phone. Consider the implementation of 'TaintCheck' on an x86 platform.[http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.83.2141&rep=rep1&type=pdf] TaintCheck performs dynamic taint analysis on a program by running the program in its own emulation environment. This allows TaintCheck to monitor and control the program’s execution at a fine-grained level. All the TaintCheck needs is the binary which it the rewrites and uses it in its own emulated environment. What this essentially means is that 'TaintCheck' is a mechanism that can perform dynamic taint analysis by performing binary rewriting at run time on an emulated envioronment. Taking this further, we can consider an implementation of TaintDroid based on similar lines. One can then envision an application in which you uploaded the 'application binary' and then TaintDroid would return a result of whether the application is safe or not. This has the advantage of being needed to run just once before installation and hence the overheads won't be much of a concern. This can even allow TaintDroid to incorporate signature based detection of 'malacious applications'.

REFERENCES

=Questions=
From Anil's questions:
* What is one source of false positives in TaintDroid? (In other words, what kind of code/data behavior leads to false alarms?)
** Answer: Some applications are making legitimate use of sensitive data; for example, Google Maps needs to know your location in order to work, and the use of this data is known by the user. TaintDroid cannot know whether the user has consented to the use of some data, and so flags it as a leak.
* What part of Android was modified for TaintDroid? Is this part of Android's kernel? Explain briefly.
** The Dalvic VM is modified. Dalvic is the java virtual machine used by Android to run user applications. Although Dalvic is a core part of the Android operating system, it is not a part of the kernel. Dalvic runs on top of the Android kernel (itself a Linux kernel) as a user process. All third party user applications run on top of Dalvic, however, so it is a sufficiently "low-level" point of the system to implement taint tracking.

Additional questions:
#
#
#

=References=
[] DENNING, D. E. [http://www.cs.georgetown.edu/~denning/infosec/lattice76.pdf A Lattice Model of Secure Information Flow].
Communications of the ACM 19, 5 (May 1976), 236–243. 
[]D CEARA, ML POTET et.al [http://tanalysis.googlecode.com/files/DumitruCeara_BSc.pdf Detecting Software Vulnerabilities Static Taint Analysis] GINP ENSIMAG GoogleCode(2009) 
[] NEWSOME,J.,AND SONG,D. [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.83.2141&rep=rep1&type=pdf Dynamic Taint Analysis for Automatic Detection, Analysis, and Signature Generation of Exploits on Commodity Software] Proceedings of the Network and Distributed System Security Symposium (NDSS 2005)

Talk:COMP 3000 Essay 2 2010 Question 8

2010-11-30T22:33:29Z

Gbint: /* Work In Progress */

Group Members

Trevor Bonesaw Malone - tmalone@connect.carleton.ca //FIRST POST!

Qi Zhang - qzhang13@connect.carleton.ca

Gregory Bint - gbint@connect.carleton.ca

Gautam Akiwate - gakiwate@connect.carleton.ca

Corey Ling - cling@connect.carleton.ca

Sarah Liske

==Relevant Sources==
*NEWSOME,J.,AND SONG,D.Dynamic Taint Analysis for Automatic Detection, Analysis, and Signature Generation of Exploits on Commodity Software. [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.83.2141&rep=rep1&type=pdf Dynamic Taint Analysis for Automatic Detection]
Seems to be THE Dynamic Taint Analysis Paper.Talks about implementation on TaintCheck. Could be also useful for critique section -[Gautam]

== Work Plan ==

As Trevor intimated, we should have clear division of work going forward. This is sort of the break down as I see it. Please edit as you think of new ideas!

* Background Concepts
** Information Flow Theory. (Implicit and Explicit Flows.) --Done[--[[User:Gautam|Gautam]] 03:54, 28 November 2010 (UTC)]
** What is dynamic taint analysis --Done[--[[User:Gautam|Gautam]] 05:07, 28 November 2010 (UTC)]
** What is the difference between dynamic and static analysis
* Research Problem
** How do we build a DTA engine for a phone?
** Why do we want to? (information misuse)
* Contribution
** How did they implement their DTA engine (Done: --[[User:Cling|Cling]] 04:50, 26 November 2010 (UTC))
** What did they find about information misuse (Done: --[[User:Cling|Cling]] 04:50, 26 November 2010 (UTC))
** Compared to the existing taint tracking approaches. [[User:Zhangqi|Zhangqi]] 07:11, 27 November 2010 (UTC) (Added something. Still looking for other examples,in progress)
** (What else should be in the contributions? Anything need fleshing out?) (Working on that now :) sliske
* Critique
**Added two paragraphs at the end of the present critique. Please incorporate it into your content as you deem fit.--[[User:Gautam|Gautam]] 09:07, 30 November 2010 (UTC) (done)
* References
** The article has 61 references! We can probably use some of them

List of information we need to find external sources for:
* History of taint analysis
* History of privacy research relating to smart phones

== Work In Progress ==

Log what you are working on *right now* so that other people don't try to do the same thing. Make sure to clear your name from here when you are done.

* Gregory Bint: Research Problem
** Need to find some history on smart phone security research for the second part.

* Gautam Akiwate: Background Concepts
** Any resources on Dynamic taint Analysis would be appreciated!

* Corey Ling: Contributions (Qi Zhang)

* Trevor Malone: Critique

* Sarah Liske: References and Questions, Clarity/Spelling.

== Some Notes from the Video ==

Tracking of privacy sensitive data through Dynamic Taint Analysis (aka. Taint Tracking). The trick is to mark private data as it sourced, and then follow those marks until (unless) they leave the phone.

Android phones run Java apps, which are compiled into DEX, and then run on top of the Dalvik VM. It is this VM that we modify so that we can support the storage and tracking of taint tags.

Taint sources
* low -bandwidth sensors
** Location
** Accelerometer
* High-bandwidth sensors
** Mic
** Camera
* Information DB
** Address book
** SMS storage
* Device ID
** IMEI
** IMSI (don't actually track this one because of false positives)
** ICC_ID
** Phone Number

Taint sink (where marked data can leave the phone)
* Network Taint Sink

Taint propagation
* ???

Taint tags are stored in memory interleaved with the variables they are tracking

Some standard Data Flow technique is used to propagate these tags, especially as one variable that is marked may be assigned to another, so now that variable needs to be tracked as well.

Tracks explicit flows of data, not implicit
To fully capture implicit flows, you need to do static analysis, which is hard with closed-source apps, and cannot be done real-time

Implicit flows are not tracked
* Implicit flows can involve "taint-scope", tracking based on conditionals in code

=== Performance ===

The goal is to create a real time tracking system, so the TaintDroid's performance impact is of some importance

14% CPU overhead
4.4% memory overhead

Macro benchmarks (to get a feel for what the phone's usability is like with TD running)
* App load: 3% (2ms)

=== Findings ===

20 out of 30 tested applications share data in a way that is not expected.

67 of 105 flagged pieces of data leaving the device had no obviously legitimate purpose (verified by the authors).

Many apps sent location data and other unique identifiers to advertising servers.

Most apps do not mention anything to the user.

=== Limitations ===

Tracks only explicit data flows.

An application *could* launder the tags off of the data, if they really wanted to hide this sort of thing from TaintDroid.

There are methods that could be used to protect against this, but they go against the goal of a light-weight, real-time tracking system. TD is not necessarily about catching truly malicious programs, but rather just those that leak information.

Why do apps take this information?
* Lazy; in the demo video, the wallpaper app seems to use the IMEI just as a ready made unique ID
* Overzealous; the developer might thing they *need* the data for something, but actually
* Ads; advertises do seem a little presumptuous in their data collection
* Spying; bosses or spouses
* Malicious;

=== QA Period ===

Q: how do we prevent a malicious app from removing a taint attribute on a file

A: TD operates a too low a level for this to be a problem; TD assumes that the native code is trusted

Q: It seems like you had a lot of false positives

A: The point of this tool was to identify privacy sensitive information as having left the phone, not whether or not a privacy violation has taken place.

Q: Now that TD is released; couldn't malicious apps use some of the methods described in the paper to get around it?

A: Well, yes, but it is not just about maliciousness, it could just laziness or over-zealous ad stuff.

==Other Information==

Hey guys, thought I would just post a generalized paragraph about our essay.

In today’s society, Smartphones are the new big thing. To me that’s what makes this paper so interesting. This paper focuses on private information in android phones and the misuse of this information. The misuse of information includes the SIM card, the ID of the device, or the phone number. TaintDroid is used on smart phones with an efficient taint tracking and analysis system. It has the ability to track sensitive data from multiple sources and examines the misuse of such data. In their study, out of 80 popular third-party applications, TaintDroid monitored that 68 applications had potential misuse of user’s private data. This tool is great for knowing with applications are safe and which are not, so your private data can remained private.

Also, we should really think of splitting up the work in some way. If some people have specific sections they would like to do lets figure that out now so we can divide the workload and get it done over the next couple of days. I don't personally care what part I'm going to have to do, so lets get this going. Any other information people wanna post feel free the more the better, even if we don't end up using it.

[[user:Tmalone|Trevor Malone]]

Talk:COMP 3000 Essay 2 2010 Question 8

2010-11-27T17:58:58Z

Gbint: /* Work In Progress */

Group Members

Trevor Bonesaw Malone - tmalone@connect.carleton.ca //FIRST POST!

Qi Zhang - qzhang13@connect.carleton.ca

Gregory Bint - gbint@connect.carleton.ca

Gautam Akiwate - gakiwate@connect.carleton.ca

Corey Ling - cling@connect.carleton.ca

==Relevant Sources==
*NEWSOME,J.,AND SONG,D.Dynamic Taint Analysis for Automatic Detection, Analysis, and Signature Generation of Exploits on Commodity Software. [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.83.2141&rep=rep1&type=pdf Dynamic Taint Analysis for Automatic Detection]
Seems to be THE Dynamic Taint Analysis Paper.Talks about implementation on TaintCheck. Could be also useful for critique section -[Gautam]

== Work Plan ==

As Trevor intimated, we should have clear division of work going forward. This is sort of the break down as I see it. Please edit as you think of new ideas!

* Background Concepts
** Information Flow Theory. (Implicit and Explicit Flows.)
** What is dynamic taint analysis
** What is the difference between dynamic and static analysis
* Research Problem
** How do we build a DTA engine for a phone?
** Why do we want to? (information misuse)
* Contribution
** How did they implement their DTA engine (Done: --[[User:Cling|Cling]] 04:50, 26 November 2010 (UTC))
** What did they find about information misuse (Done: --[[User:Cling|Cling]] 04:50, 26 November 2010 (UTC))
** Compared to the existing taint tracking approaches. [[User:Zhangqi|Zhangqi]] 07:11, 27 November 2010 (UTC) (Added something. Still looking for other examples,in progress)
** (What else should be in the contributions? Anything need fleshing out?)
* Critique
* References
** The article has 61 references! We can probably use some of them

List of information we need to find external sources for:
* History of taint analysis
* History of privacy research relating to smart phones

== Work In Progress ==

Log what you are working on *right now* so that other people don't try to do the same thing. Make sure to clear your name from here when you are done.

* Gregory Bint: Research Problem
** I've got what you will hopefully find to be a reasonable introduction to the problem. I've tried to ask mostly questions here, expecting the details to be covered in Background Concepts and in Contributions.
** I'm going to try to find some history on smart phone security research for the second part.

* Gautam Akiwate: Background Concepts
** Any resources on Dynamic taint Analysis would be appreciated!

* Corey Ling: Contributions (Qi Zhang)

* Trevor Malone: Critique

== Some Notes from the Video ==

Tracking of privacy sensitive data through Dynamic Taint Analysis (aka. Taint Tracking). The trick is to mark private data as it sourced, and then follow those marks until (unless) they leave the phone.

Android phones run Java apps, which are compiled into DEX, and then run on top of the Dalvik VM. It is this VM that we modify so that we can support the storage and tracking of taint tags.

Taint sources
* low -bandwidth sensors
** Location
** Accelerometer
* High-bandwidth sensors
** Mic
** Camera
* Information DB
** Address book
** SMS storage
* Device ID
** IMEI
** IMSI (don't actually track this one because of false positives)
** ICC_ID
** Phone Number

Taint sink (where marked data can leave the phone)
* Network Taint Sink

Taint propagation
* ???

Taint tags are stored in memory interleaved with the variables they are tracking

Some standard Data Flow technique is used to propagate these tags, especially as one variable that is marked may be assigned to another, so now that variable needs to be tracked as well.

Tracks explicit flows of data, not implicit
To fully capture implicit flows, you need to do static analysis, which is hard with closed-source apps, and cannot be done real-time

Implicit flows are not tracked
* Implicit flows can involve "taint-scope", tracking based on conditionals in code

=== Performance ===

The goal is to create a real time tracking system, so the TaintDroid's performance impact is of some importance

14% CPU overhead
4.4% memory overhead

Macro benchmarks (to get a feel for what the phone's usability is like with TD running)
* App load: 3% (2ms)

=== Findings ===

20 out of 30 tested applications share data in a way that is not expected.

67 of 105 flagged pieces of data leaving the device had no obviously legitimate purpose (verified by the authors).

Many apps sent location data and other unique identifiers to advertising servers.

Most apps do not mention anything to the user.

=== Limitations ===

Tracks only explicit data flows.

An application *could* launder the tags off of the data, if they really wanted to hide this sort of thing from TaintDroid.

There are methods that could be used to protect against this, but they go against the goal of a light-weight, real-time tracking system. TD is not necessarily about catching truly malicious programs, but rather just those that leak information.

Why do apps take this information?
* Lazy; in the demo video, the wallpaper app seems to use the IMEI just as a ready made unique ID
* Overzealous; the developer might thing they *need* the data for something, but actually
* Ads; advertises do seem a little presumptuous in their data collection
* Spying; bosses or spouses
* Malicious;

=== QA Period ===

Q: how do we prevent a malicious app from removing a taint attribute on a file

A: TD operates a too low a level for this to be a problem; TD assumes that the native code is trusted

Q: It seems like you had a lot of false positives

A: The point of this tool was to identify privacy sensitive information as having left the phone, not whether or not a privacy violation has taken place.

Q: Now that TD is released; couldn't malicious apps use some of the methods described in the paper to get around it?

A: Well, yes, but it is not just about maliciousness, it could just laziness or over-zealous ad stuff.

==Other Information==

Hey guys, thought I would just post a generalized paragraph about our essay.

In today’s society, Smartphones are the new big thing. To me that’s what makes this paper so interesting. This paper focuses on private information in android phones and the misuse of this information. The misuse of information includes the SIM card, the ID of the device, or the phone number. TaintDroid is used on smart phones with an efficient taint tracking and analysis system. It has the ability to track sensitive data from multiple sources and examines the misuse of such data. In their study, out of 80 popular third-party applications, TaintDroid monitored that 68 applications had potential misuse of user’s private data. This tool is great for knowing with applications are safe and which are not, so your private data can remained private.

Also, we should really think of splitting up the work in some way. If some people have specific sections they would like to do lets figure that out now so we can divide the workload and get it done over the next couple of days. I don't personally care what part I'm going to have to do, so lets get this going. Any other information people wanna post feel free the more the better, even if we don't end up using it.

[[user:Tmalone|Trevor Malone]]

Talk:COMP 3000 Essay 2 2010 Question 8

2010-11-23T01:02:04Z

Gbint: /* Work In Progress */

Group Members

Trevor Bonesaw Malone - tmalone@connect.carleton.ca //FIRST POST!

Qi Zhang - qzhang13@connect.carleton.ca

Gregory Bint - gbint@connect.carleton.ca

Gautam Akiwate - gakiwate@connect.carleton.ca

==Relevant Sources==
*NEWSOME,J.,AND SONG,D.Dynamic Taint Analysis for Automatic Detection, Analysis, and Signature Generation of Exploits on Commodity Software. [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.83.2141&rep=rep1&type=pdf Dynamic Taint Analysis for Automatic Detection]
Seems to be THE Dynamic Taint Analysis Paper.Talks about implementation on TaintCheck. Could be also useful for critique section -[Gautam]

== Work Plan ==

As Trevor intimated, we should have clear division of work going forward. This is sort of the break down as I see it. Please edit as you think of new ideas!

* Background Concepts
** Information Flow Theory. (Implicit and Explicit Flows.)
** What is dynamic taint analysis
** What is the difference between dynamic and static analysis
* Research Problem
** How do we build a DTA engine for a phone?
** Why do we want to? (information misuse)
* Contribution
** How did they implement their DTA engine
** What did they find about information misuse
* Critique
* References
** The article has 61 references! We can probably use some of them

List of information we need to find external sources for:
* History of taint analysis
* History of privacy research relating to smart phones

== Work In Progress ==

Log what you are working on *right now* so that other people don't try to do the same thing. Make sure to clear your name from here when you are done.

* Gregory Bint: Research Problem
** I've got what you will hopefully find to be a reasonable introduction to the problem. I've tried to ask mostly questions here, expecting the details to be covered in Background Concepts and in Contributions.
** I'm going to try to find some history on smart phone security research for the second part.
** Should we move the Research Problem *above* Background Concepts? It might serve as a better lead in that way.

* Gautam Akiwate: Background Concepts
** Any resources on Dynamic taint Analysis would be appreciated!

* Qi Zhang: Contributions

== Some Notes from the Video ==

Tracking of privacy sensitive data through Dynamic Taint Analysis (aka. Taint Tracking). The trick is to mark private data as it sourced, and then follow those marks until (unless) they leave the phone.

Android phones run Java apps, which are compiled into DEX, and then run on top of the Dalvik VM. It is this VM that we modify so that we can support the storage and tracking of taint tags.

Taint sources
* low -bandwidth sensors
** Location
** Accelerometer
* High-bandwidth sensors
** Mic
** Camera
* Information DB
** Address book
** SMS storage
* Device ID
** IMEI
** IMSI (don't actually track this one because of false positives)
** ICC_ID
** Phone Number

Taint sink (where marked data can leave the phone)
* Network Taint Sink

Taint propagation
* ???

Taint tags are stored in memory interleaved with the variables they are tracking

Some standard Data Flow technique is used to propagate these tags, especially as one variable that is marked may be assigned to another, so now that variable needs to be tracked as well.

Tracks explicit flows of data, not implicit
To fully capture implicit flows, you need to do static analysis, which is hard with closed-source apps, and cannot be done real-time

Implicit flows are not tracked
* Implicit flows can involve "taint-scope", tracking based on conditionals in code

=== Performance ===

The goal is to create a real time tracking system, so the TaintDroid's performance impact is of some importance

14% CPU overhead
4.4% memory overhead

Macro benchmarks (to get a feel for what the phone's usability is like with TD running)
* App load: 3% (2ms)

=== Findings ===

20 out of 30 tested applications share data in a way that is not expected.

67 of 105 flagged pieces of data leaving the device had no obviously legitimate purpose (verified by the authors).

Many apps sent location data and other unique identifiers to advertising servers.

Most apps do not mention anything to the user.

=== Limitations ===

Tracks only explicit data flows.

An application *could* launder the tags off of the data, if they really wanted to hide this sort of thing from TaintDroid.

There are methods that could be used to protect against this, but they go against the goal of a light-weight, real-time tracking system. TD is not necessarily about catching truly malicious programs, but rather just those that leak information.

Why do apps take this information?
* Lazy; in the demo video, the wallpaper app seems to use the IMEI just as a ready made unique ID
* Overzealous; the developer might thing they *need* the data for something, but actually
* Ads; advertises do seem a little presumptuous in their data collection
* Spying; bosses or spouses
* Malicious;

=== QA Period ===

Q: how do we prevent a malicious app from removing a taint attribute on a file

A: TD operates a too low a level for this to be a problem; TD assumes that the native code is trusted

Q: It seems like you had a lot of false positives

A: The point of this tool was to identify privacy sensitive information as having left the phone, not whether or not a privacy violation has taken place.

Q: Now that TD is released; couldn't malicious apps use some of the methods described in the paper to get around it?

A: Well, yes, but it is not just about maliciousness, it could just laziness or over-zealous ad stuff.

==Other Information==

Hey guys, thought I would just post a generalized paragraph about our essay.

In today’s society, Smartphones are the new big thing. To me that’s what makes this paper so interesting. This paper focuses on private information in android phones and the misuse of this information. The misuse of information includes the SIM card, the ID of the device, or the phone number. TaintDroid is used on smart phones with an efficient taint tracking and analysis system. It has the ability to track sensitive data from multiple sources and examines the misuse of such data. In their study, out of 80 popular third-party applications, TaintDroid monitored that 68 applications had potential misuse of user’s private data. This tool is great for knowing with applications are safe and which are not, so your private data can remained private.

Also, we should really think of splitting up the work in some way. If some people have specific sections they would like to do lets figure that out now so we can divide the workload and get it done over the next couple of days. I don't personally care what part I'm going to have to do, so lets get this going. Any other information people wanna post feel free the more the better, even if we don't end up using it.

[[user:Tmalone|Trevor Malone]]

COMP 3000 Essay 2 2010 Question 8

2010-11-23T00:57:05Z

Gbint: /* Research problem */ I've tried to ask mostly questions here, with the details supposedly in Background concepts or in Contribution

=Paper=

<big><big>TaintDroid: An Information-Flow Tracking System for Realtime Privacy Monitoring on Smartphones</big></big>

Authors:
* William Enck, ''The Pennsylvania State University''
* Peter Gilbert, ''Duke University''
* Byung-Gon Chun, ''Intel Labs''
* Landon P. Cox, ''Duke University''
* Jaeyeon Jung, ''Intel Labs''
* Patrick McDaniel, ''The Pennsylvania State University''
* Anmol N. Sheth, ''Intel Labs''

Official Website: http://www.appanalysis.org/

Direct Link to Paper: http://appanalysis.org/tdroid10.pdf

Video demonstration of TaintDroid in action: http://www.youtube.com/watch?v=qnLujX1Dw4Y

=Background Concepts=
Explain briefly the background concepts and ideas that your fellow classmates will need to know first in order to understand your assigned paper.

* Background on Information Flow Theory. Explicit and Implicit Flow.
* Background on the taint data tracking method, how it has been used in other systems (i.e. not phones)
* A reader's digest version of any new articles about this kind of security vulnerability on phones, on apps that collect more personal data than users would expect.

=Research problem=

In today’s society, smartphones are the new big thing. Smartphones, by their nature, are linked into many private details of our lives, including not only classic data like our contact list, but new kinds of data unique to smartphones, such as location data. Except for the odd tunnel or elevator, these phones are constantly connected to the internet. Smartphones also have the ability to download and run third party applications; indeed, this is why we call them "smart". When you combine third party applications with an internet connection, you suddenly find yourself unsure of how your data is being used, that is, what is to stop a third party application from disseminating our private information? As it turns out, very little.

A telling example of this is a wallpaper application that sends your phone number back to the developer. Once the app is running on your phone, it can typically access any of the information on your phone, and it is not necessarily clear when it has done so, or what it is doing with it.

The authors of this paper set out to try to understand what kind of information is being collected and where that information is being sent, and in order to do that, they first needed to build a means of tracking that information.

The strategy they chose is called Dynamic Taint Analysis, sometimes called Taint Tracking. The basic idea being to mark (''taint'') sensitive information at its source, and to then follow that mark as it moves through a system. In the context of this paper, if ever we should see marked data leave the network interface of the phone, then we know that some sensitive information has been disseminated.

There are many difficulties associated with implementing such a system on a smartphone. Their design goals were to create a light-weight, minimal overhead, real-time tracking system that runs directly on a real phone, with real applications. To be really useful, the tracking system must not impact the user experience too heavily.

Some of the difficulties include
* Smart phones are resource constrained. Processing power and memory are limited, and any processing that we do perform will consume battery power. If the tracking system is to be real-time, and for the phone to be considered "usable" by the end user, the system must be truly light weight.
* Third party applications arrive in a compiled format; we cannot analyze their source code.
* Applications may do complex things with the sensitive data. It is unlikely that the application will simply read a location from the GPS and dump it straight out over the network. More likely is that the application will use that data in someway, or combine it with other data, before it is sent. We need to be able to track sensitive data throughout this entire process if we hope to perform any useful analysis.
* Applications can share information with other applications, meaning that our tracking has to work across multiple processes.
* The tracking must operate on a real phone, not a simulated one. With a simulated system, where we control the virtual hardware and memory, we can be certain that we can see everything that an application might do. On a real device, how can we get "low enough" to see everything the applications do?

How does this problem relate to past related work?

=Contribution=
What are the research contribution(s) of this work? Specifically, what are the key research results, and what do they mean? (What was implemented? Why is it any better than what came before?)

=Critique=
What is good and not-so-good about this paper? You may discuss both the style and content; be sure to ground your discussion with specific references. Simple assertions that something is good or bad is not enough - you must explain why.

=References=
You will almost certainly have to refer to other resources; please cite these resources in the style of citation of the papers assigned (inlined numbered references). Place your bibliographic entries in this section.

COMP 3000 Essay 2 2010 Question 8

2010-11-22T23:36:43Z

Gbint: /* Research problem */

COMP 3000 Essay 2 2010 Question 8

2010-11-22T21:59:00Z

Gbint: /* Research problem */ rough work; jotting out ideas

Talk:COMP 3000 Essay 2 2010 Question 8

2010-11-22T21:56:39Z

Gbint: work plan

Group Members

Trevor Bonesaw Malone - tmalone@connect.carleton.ca //FIRST POST!

Qi Zhang - qzhang13@connect.carleton.ca

Gregory Bint - gbint@connect.carleton.ca

Gautam Akiwate - gakiwate@connect.carleton.ca

== Work Plan ==

As Trevor intimated, we should have clear division of work going forward. This is sort of the break down as I see it. Please edit as you think of new ideas!

* Background Concepts
** What is dynamic taint analysis
** What is the difference between dynamic and static analysis
* Research Problem
** How do we build a DTA engine for a phone?
** Why do we want to? (information misuse)
* Contribution
** How did they implement their DTA engine
** What did they find about information misuse
* Critique
* References
** The article has 61 references! We can probably use some of them

List of information we need to find external sources for:
* History of taint analysis
* History of privacy research relating to smart phones

== Work In Progress ==

Log what you are working on *right now* so that other people don't try to do the same thing. Make sure to clear your name from here when you are done.

* Gregory Bint: Research Problem
** If someone wants to find some sources for some of the questions I'm asking, that would be helpful!

== Some Notes from the Video ==

Tracking of privacy sensitive data through Dynamic Taint Analysis (aka. Taint Tracking). The trick is to mark private data as it sourced, and then follow those marks until (unless) they leave the phone.

Android phones run Java apps, which are compiled into DEX, and then run on top of the Dalvik VM. It is this VM that we modify so that we can support the storage and tracking of taint tags.

Taint sources
* low -bandwidth sensors
** Location
** Accelerometer
* High-bandwidth sensors
** Mic
** Camera
* Information DB
** Address book
** SMS storage
* Device ID
** IMEI
** IMSI (don't actually track this one because of false positives)
** ICC_ID
** Phone Number

Taint sink (where marked data can leave the phone)
* Network Taint Sink

Taint propagation
* ???

Taint tags are stored in memory interleaved with the variables they are tracking

Some standard Data Flow technique is used to propagate these tags, especially as one variable that is marked may be assigned to another, so now that variable needs to be tracked as well.

Tracks explicit flows of data, not implicit
To fully capture implicit flows, you need to do static analysis, which is hard with closed-source apps, and cannot be done real-time

Implicit flows are not tracked
* Implicit flows can involve "taint-scope", tracking based on conditionals in code

=== Performance ===

The goal is to create a real time tracking system, so the TaintDroid's performance impact is of some importance

14% CPU overhead
4.4% memory overhead

Macro benchmarks (to get a feel for what the phone's usability is like with TD running)
* App load: 3% (2ms)

=== Findings ===

20 out of 30 tested applications share data in a way that is not expected.

67 of 105 flagged pieces of data leaving the device had no obviously legitimate purpose (verified by the authors).

Many apps sent location data and other unique identifiers to advertising servers.

Most apps do not mention anything to the user.

=== Limitations ===

Tracks only explicit data flows.

An application *could* launder the tags off of the data, if they really wanted to hide this sort of thing from TaintDroid.

There are methods that could be used to protect against this, but they go against the goal of a light-weight, real-time tracking system. TD is not necessarily about catching truly malicious programs, but rather just those that leak information.

Why do apps take this information?
* Lazy; in the demo video, the wallpaper app seems to use the IMEI just as a ready made unique ID
* Overzealous; the developer might thing they *need* the data for something, but actually
* Ads; advertises do seem a little presumptuous in their data collection
* Spying; bosses or spouses
* Malicious;

=== QA Period ===

Q: how do we prevent a malicious app from removing a taint attribute on a file

A: TD operates a too low a level for this to be a problem; TD assumes that the native code is trusted

Q: It seems like you had a lot of false positives

A: The point of this tool was to identify privacy sensitive information as having left the phone, not whether or not a privacy violation has taken place.

Q: Now that TD is released; couldn't malicious apps use some of the methods described in the paper to get around it?

A: Well, yes, but it is not just about maliciousness, it could just laziness or over-zealous ad stuff.

==Other Information==

Hey guys, thought I would just post a generalized paragraph about our essay.

In today’s society, Smartphones are the new big thing. To me that’s what makes this paper so interesting. This paper focuses on private information in android phones and the misuse of this information. The misuse of information includes the SIM card, the ID of the device, or the phone number. TaintDroid is used on smart phones with an efficient taint tracking and analysis system. It has the ability to track sensitive data from multiple sources and examines the misuse of such data. In their study, out of 80 popular third-party applications, TaintDroid monitored that 68 applications had potential misuse of user’s private data. This tool is great for knowing with applications are safe and which are not, so your private data can remained private.

Also, we should really think of splitting up the work in some way. If some people have specific sections they would like to do lets figure that out now so we can divide the workload and get it done over the next couple of days. I don't personally care what part I'm going to have to do, so lets get this going. Any other information people wanna post feel free the more the better, even if we don't end up using it.

[[user:Tmalone|Trevor Malone]]

Talk:COMP 3000 Essay 2 2010 Question 8

2010-11-20T04:33:14Z

Gbint: /* QA Period */ formatting

Group Members

Trevor Bonesaw Malone - tmalone@connect.carleton.ca //FIRST POST!

Qi Zhang - qzhang13@connect.carleton.ca

Gregory Bint - gbint@connect.carleton.ca

Gautam Akiwate - gakiwate@connect.carleton.ca

== Some Notes from the Video ==

Tracking of privacy sensitive data through Dynamic Taint Analysis (aka. Taint Tracking). The trick is to mark private data as it sourced, and then follow those marks until (unless) they leave the phone.

Android phones run Java apps, which are compiled into DEX, and then run on top of the Dalvik VM. It is this VM that we modify so that we can support the storage and tracking of taint tags.

Taint sources
* low -bandwidth sensors
** Location
** Accelerometer
* High-bandwidth sensors
** Mic
** Camera
* Information DB
** Address book
** SMS storage
* Device ID
** IMEI
** IMSI (don't actually track this one because of false positives)
** ICC_ID
** Phone Number

Taint sink (where marked data can leave the phone)
* Network Taint Sink

Taint propagation
* ???

Taint tags are stored in memory interleaved with the variables they are tracking

Some standard Data Flow technique is used to propagate these tags, especially as one variable that is marked may be assigned to another, so now that variable needs to be tracked as well.

Tracks explicit flows of data, not implicit
To fully capture implicit flows, you need to do static analysis, which is hard with closed-source apps, and cannot be done real-time

Implicit flows are not tracked
* Implicit flows can involve "taint-scope", tracking based on conditionals in code

=== Performance ===

The goal is to create a real time tracking system, so the TaintDroid's performance impact is of some importance

14% CPU overhead
4.4% memory overhead

Macro benchmarks (to get a feel for what the phone's usability is like with TD running)
* App load: 3% (2ms)

=== Findings ===

20 out of 30 tested applications share data in a way that is not expected.

67 of 105 flagged pieces of data leaving the device had no obviously legitimate purpose (verified by the authors).

Many apps sent location data and other unique identifiers to advertising servers.

Most apps do not mention anything to the user.

=== Limitations ===

Tracks only explicit data flows.

An application *could* launder the tags off of the data, if they really wanted to hide this sort of thing from TaintDroid.

There are methods that could be used to protect against this, but they go against the goal of a light-weight, real-time tracking system. TD is not necessarily about catching truly malicious programs, but rather just those that leak information.

Why do apps take this information?
* Lazy; in the demo video, the wallpaper app seems to use the IMEI just as a ready made unique ID
* Overzealous; the developer might thing they *need* the data for something, but actually
* Ads; advertises do seem a little presumptuous in their data collection
* Spying; bosses or spouses
* Malicious;

=== QA Period ===

Q: how do we prevent a malicious app from removing a taint attribute on a file

A: TD operates a too low a level for this to be a problem; TD assumes that the native code is trusted

Q: It seems like you had a lot of false positives

A: The point of this tool was to identify privacy sensitive information as having left the phone, not whether or not a privacy violation has taken place.

Q: Now that TD is released; couldn't malicious apps use some of the methods described in the paper to get around it?

A: Well, yes, but it is not just about maliciousness, it could just laziness or over-zealous ad stuff.

Talk:COMP 3000 Essay 2 2010 Question 8

2010-11-20T04:32:44Z

Gbint: my notes from the video; please add!

Group Members

Trevor Bonesaw Malone - tmalone@connect.carleton.ca //FIRST POST!

Qi Zhang - qzhang13@connect.carleton.ca

Gregory Bint - gbint@connect.carleton.ca

Gautam Akiwate - gakiwate@connect.carleton.ca

== Some Notes from the Video ==

Tracking of privacy sensitive data through Dynamic Taint Analysis (aka. Taint Tracking). The trick is to mark private data as it sourced, and then follow those marks until (unless) they leave the phone.

Android phones run Java apps, which are compiled into DEX, and then run on top of the Dalvik VM. It is this VM that we modify so that we can support the storage and tracking of taint tags.

Taint sources
* low -bandwidth sensors
** Location
** Accelerometer
* High-bandwidth sensors
** Mic
** Camera
* Information DB
** Address book
** SMS storage
* Device ID
** IMEI
** IMSI (don't actually track this one because of false positives)
** ICC_ID
** Phone Number

Taint sink (where marked data can leave the phone)
* Network Taint Sink

Taint propagation
* ???

Taint tags are stored in memory interleaved with the variables they are tracking

Some standard Data Flow technique is used to propagate these tags, especially as one variable that is marked may be assigned to another, so now that variable needs to be tracked as well.

Tracks explicit flows of data, not implicit
To fully capture implicit flows, you need to do static analysis, which is hard with closed-source apps, and cannot be done real-time

Implicit flows are not tracked
* Implicit flows can involve "taint-scope", tracking based on conditionals in code

=== Performance ===

The goal is to create a real time tracking system, so the TaintDroid's performance impact is of some importance

14% CPU overhead
4.4% memory overhead

Macro benchmarks (to get a feel for what the phone's usability is like with TD running)
* App load: 3% (2ms)

=== Findings ===

20 out of 30 tested applications share data in a way that is not expected.

67 of 105 flagged pieces of data leaving the device had no obviously legitimate purpose (verified by the authors).

Many apps sent location data and other unique identifiers to advertising servers.

Most apps do not mention anything to the user.

=== Limitations ===

Tracks only explicit data flows.

An application *could* launder the tags off of the data, if they really wanted to hide this sort of thing from TaintDroid.

There are methods that could be used to protect against this, but they go against the goal of a light-weight, real-time tracking system. TD is not necessarily about catching truly malicious programs, but rather just those that leak information.

Why do apps take this information?
* Lazy; in the demo video, the wallpaper app seems to use the IMEI just as a ready made unique ID
* Overzealous; the developer might thing they *need* the data for something, but actually
* Ads; advertises do seem a little presumptuous in their data collection
* Spying; bosses or spouses
* Malicious;

=== QA Period ===

Q: how do we prevent a malicious app from removing a taint attribute on a file
A: TD operates a too low a level for this to be a problem; TD assumes that the native code is trusted

Q: It seems like you had a lot of false positives
A: The point of this tool was to identify privacy sensitive information as having left the phone, not whether or not a privacy violation has taken place.

Q: Now that TD is released; couldn't malicious apps use some of the methods described in the paper to get around it?
A: Well, yes, but it is not just about maliciousness, it could just laziness or over-zealous ad stuff.

COMP 3000 Essay 2 2010 Question 8

2010-11-20T01:34:43Z

Gbint: added title, author info

COMP 3000 Essay 2 2010 Question 8

2010-11-18T02:20:25Z

Gbint:

=Paper=
The paper's title, authors, and their affiliations. Include a link to the paper and any particularly helpful supplementary information.

=Background Concepts=
Explain briefly the background concepts and ideas that your fellow classmates will need to know first in order to understand your assigned paper.

* Background on the taint data tracking method, how it has been used in other systems (i.e. not phones)
* A reader's digest version of any new articles about this kind of security vulnerability on phones, on apps that collect more personal data than users would expect.

=Research problem=
What is the research problem being addressed by the paper? How does this problem relate to past related work?

=Contribution=
What are the research contribution(s) of this work? Specifically, what are the key research results, and what do they mean? (What was implemented? Why is it any better than what came before?)

=Critique=
What is good and not-so-good about this paper? You may discuss both the style and content; be sure to ground your discussion with specific references. Simple assertions that something is good or bad is not enough - you must explain why.

=References=
You will almost certainly have to refer to other resources; please cite these resources in the style of citation of the papers assigned (inlined numbered references). Place your bibliographic entries in this section.

Talk:COMP 3000 Essay 2 2010 Question 8

2010-11-18T02:17:24Z

Gbint:

Group Members

Trevor Bonesaw Malone - tmalone@connect.carleton.ca //FIRST POST!

Qi Zhang - qzhang13@connect.carleton.ca

Gregory Bint - gbint@connect.carleton.ca

COMP 3000 Lab 4 2010

2010-11-02T14:57:17Z

Gbint: how to add a custom grub menu

All of the following should be done with an Ubuntu 10.04 distribution or equivalent. We recommend experimenting in a virtual environment because some of the exercises could make your system unbootable. (In fact, take a snapshot of your working system before starting these exercises so you can easily revert.)

==Questions==

# Change the grub command line at boot to limit the total available RAM to 256M. You'll need to get to select an entry and edit it from within grub.
# Add a new grub menu item which limits the standard kernel to 256M.
# Add a second virtual disk and make it bootable: put the kernel and initial ram disk on it and then install grub. Can you boot off of this disk? What does it do?
# Examine the standard kernel's initial ram disk (initrd). What program is first run in this environment? What does it do?
# Modify the standard initial RAM disk so it pauses for 10 seconds and prints a message to the console on boot.
# What programs does upstart start on boot?

==Hints==

Please add your hints below to help your fellow students!

=== Pro-tip ===
sudo update-grub (important!!! guess what it does!!!)

===Kernel command line options===

===GRUB configuration===

*On Ubuntu the user configuration is stored in /etc/default/grub, while system grub configuration is in /etc/grub.d. The main grub files are stored in /boot/grub. You can update grub's config with the update-grub command.
*to limit RAM, go to /etc/default/grub, and change default boot options so that mem=256M is included.

=== Adding a new grub menu item ===

You need to edit the /etc/grub.d/40_custom file. You can add custom menu items there using:

menuentry "my menu item name"
{
boot parameters go here
}

To find out what boot parameters you need, reboot your system and look at the grub boot menu (hold down shift if your grub menu is being hidden by default, or comment out the GRUB_HIDDEN_* items in /etc/default/grub)

Once at the grub menu, select the default Linux boot item (the first one), and press 'e'; What you will then be shown is the grub script necessary to start Linux. To make your own menu item, you want to add all of these things, exactly as seen, plus any extra kernel parameters (like mem=256M)

===How GRUB works===

===Making a disk bootable===

===Examining RAM disks===

*Ubuntu (Debian) store initial RAM disks in the cpio format. 'zcat <file> | cpio -i' will extract its contents.

===Upstart/init===

Upstart "jobs" are config (.conf) files in /etc/init that require one of two options an "exec" line or a "script"

The exec line allows the upstart to just simply execute a script elsewhere, while script allows you to shell script in the upstart job.

A special upstart job is rc.conf which maintains the original runlevel init.d scripts. You will see that rc.conf simply executes all the /etc/init.d scripts.

See more on upstart jobs at http://upstart.ubuntu.com/getting-started.html

For those using the newer versions of Ubuntu and thus using Grub2
https://help.ubuntu.com/community/Grub2

Talk:COMP 3000 Midterm review 2010

2010-10-20T15:16:24Z

Gbint: discussing 11 and 12

--[[User:Gbint|Gbint]] 15:16, 20 October 2010 (UTC) I think that questions 11 and 12 are the most foreign concepts to me, as I have never had the opportunity to be around an object-based or database-based file system. Both of these ideas center around providing storage at a level more intelligent than just a block, with meta data and permissions sort of built-in rather than bolted on as an afterthought (a mature afterthought, sure, but its all still just blocks).

Current OFS and DbFS systems seem to just be an abstraction on top of blocks, though, even if it is implemented on the HDD controller. Do you think we will see storage systems where the basic unit of storage is something other a block?

COMP 3000 Essay 1 2010 Question 9

2010-10-10T22:52:29Z

Gbint: better reference for end-to-end zfs paper

=Question=

What requirements distinguish the Zettabyte File System (ZFS) from traditional file systems? How are those requirements realized in ZFS, and how do other operating systems address those same requirements? (Please discuss legacy, current, and in-development systems.)

=Answer=

ZFS was developed by Sun Microsystems (now owned by Oracle) as a server class file systems. This differs from most file systems which were developed as desktop file systems that could be used by servers. With the server being the target for the file system particular attention was paid to data integrity, size and speed.

One of the most significant ways in which the ZFS differs from traditional file systems is the level of abstraction. While a traditional file system abstracts away the physical properties of the media upon which it lies i.e. hard disk, flash drive, CD-ROM, etc. ZFS abstracts away if the file system lives one or many different pieces of hardware or media. Examples include a single hard drive, an array of hardrives, a number of hard drives on non co-located systems.

One of the mechanisms that allows this abstraction is that the volume manager which is normally a program separate from the file system in traditional file systems is moved into ZFS.

ZFS is a 128-bit file system allowing this allows addressing of 2128 bytes of storage.

=== Major Features of ZFS ===

====Physical Layer Abstraction====

* volume management and file system all in one
* file systems on top of zpools on top of vdevs on top of physical devices
* file systems easily and often span over many physical devices.
* ridiculous capacity

====Data Integrity====

At the lowest level, ZFS uses checksums for every block of data that is written to disk. The checksum is checked whenever data is read to ensure that data has not been corrupted in some way. The idea is that if either the block or the checksum is corrupted, then recalculating the checksum for the block will result in a mismatch between the calculated and stored checksums. It is possible that both the block and checksum record could be corrupted, but the probability of the corruption being such that the corrupted block's checksum matches the corrupted checksum is exceptionally low.

In the event that a bad checksum is found, replication of data, in the form of "Ditto Blocks" provide an opportunity for recovery. A block pointer in ZFS is actually capable of pointing to multiple blocks, each of which contains duplicate data. By default, duplicate blocks are only stored for file system metadata, but this can be expanded to user data blocks as well. When a bad checksum is read, ZFS is able to follow one of the other pointers in the block pointer to hopefully find a healthy block.

RAID setups are particularly well suited to ZFS, since there is already an abstraction between the physical storage and the zpools. Besides protecting from outright total disk failure, if a bad checksum is found, there is the possibility that one of the alternate disks has a healthy version. If these errors accumulate, it can signal an impending drive failure. When a drive does fail, some of our tolerance for data loss is consumed; that is, the system is operating at less than 100% redundancy (however that is defined for the system at hand). To address this, ZFS supports "hot spares", idle drives that can be brought online automatically when another drive fails so that full redundancy can be rebuilt with minimal delay, hopefully in time for the next drive failure.

With block-by-block data integrity well in hand, ZFS also employs a transactional update model to ensure that higher level data structures remain consistent. Rather than use a journal to allow for quick consistency checking in the event of a system crash, ZFS uses a copy-on-write model. New disk structures are written out in a detached state. Once these structures have been written and checked, then they are connected to the existing disk structures in one atomic write, with the structures they replace becoming disconnected.

At the user level, ZFS supports file-system snapshots. Essentially, a clone of the entire file system at a certain point in time is created. In the event of accidental file deletion, a user can access an older version out of a recent snapshot.

====Data Deduplication====

Data Deduplication is a method of interfile storage compression, based around the idea of storing any one block of unique data only once physically, and logically linking that block to each file that contains that data. Effective use of data deduplication can reduce the space and power requirements of physical storage, but only if you have data that lends itself to deduplication.

Data Deduplication schemes are typically implemented using hash-tables, and can be applied to whole files, sub files (blocks), or as a patch set. There is an inherit trade off between the granularity of your deduplication algorithm and the resources needed to implement it. In general, as you consider smaller blocks of data for deduplication, you increase your "fold factor", that is, the difference between the logical storage provided vs. the physical storage needed. At the same time, however, smaller blocks means more hash table overhead and more CPU time needed for deduplication and for reconstruction.

The actual analysis and deduplication of incoming files can occur in-band or out-of-band. In-band deduplication means that the file is analyzed as it arrives at the storage server, and written to disk in its already compressed state. While this method requires the least over all storage capacity, resource constraints of the server may limit the speed at which new data can be ingested. In particular, the server must have enough memory to store the entire deduplication hash table in memory for fast comparisons. With out-of-band deduplication, inbound files are written to disk without any analysis (so, in the traditional way). A background process analyzes these files at a later time to perform the compression. This method means higher overall disk I/O is needed, which can be a problem if the disk (or disk array) is already at I/O capacity.

In the case of ZFS, which is typically hosted as a server-side file system, the server itself performs all of the deduplication and reconstruction; the entire process is transparent to the client. ZFS assumes that it is running on a highly multi-threaded operating system and that CPU cycles are in greater abundance than disk I/O cycles, and thus performs the deduplication in-band.

== References ==

* Mandagere, N., Zhou, P., Smith, M. A., and Uttamchandani, S. 2008. [http://portal.acm.org.proxy.library.carleton.ca/citation.cfm?id=1462739 Demystifying data deduplication]. In Proceedings of the ACM/IFIP/USENIX Middleware '08 Conference Companion (Leuven, Belgium, December 01 - 05, 2008). Companion '08. ACM, New York, NY, 12-17.

* Geer, D.; , [http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4712493 "Reducing the Storage Burden via Data Deduplication,"] Computer , vol.41, no.12, pp.15-17, Dec. 2008

* Bonwick, J.; [http://blogs.sun.com/bonwick/en_US/entry/zfs_dedup ZFS Deduplication]. Jeff Bonwick's Blog. November 2, 2009.

* Andrew Li, Department of Computing Macquarie University, Zettabyte File System Autopsy: Digital Crime Scene Investigation for Zettabyte File System

* Zhang, Yupu and Rajimwale, Abhishek and Arpaci-Dusseau, Andrea C. and Arpaci-Dusseau, Remzi H.; [http://www.usenix.org/events/fast10/tech/full_papers/zhang.pdf End-to-end Data Integrity for File Systems: A ZFS Case Study]. FAST'10: Proceedings of the 8th USENIX conference on File and storage technologies. USENIX Association, Berkley, CA, USA.

COMP 3000 Essay 1 2010 Question 9

2010-10-10T22:48:50Z

Gbint: added reference

=Question=

What requirements distinguish the Zettabyte File System (ZFS) from traditional file systems? How are those requirements realized in ZFS, and how do other operating systems address those same requirements? (Please discuss legacy, current, and in-development systems.)

=Answer=

ZFS was developed by Sun Microsystems (now owned by Oracle) as a server class file systems. This differs from most file systems which were developed as desktop file systems that could be used by servers. With the server being the target for the file system particular attention was paid to data integrity, size and speed.

One of the most significant ways in which the ZFS differs from traditional file systems is the level of abstraction. While a traditional file system abstracts away the physical properties of the media upon which it lies i.e. hard disk, flash drive, CD-ROM, etc. ZFS abstracts away if the file system lives one or many different pieces of hardware or media. Examples include a single hard drive, an array of hardrives, a number of hard drives on non co-located systems.

One of the mechanisms that allows this abstraction is that the volume manager which is normally a program separate from the file system in traditional file systems is moved into ZFS.

ZFS is a 128-bit file system allowing this allows addressing of 2128 bytes of storage.

=== Major Features of ZFS ===

====Physical Layer Abstraction====

* volume management and file system all in one
* file systems on top of zpools on top of vdevs on top of physical devices
* file systems easily and often span over many physical devices.
* ridiculous capacity

====Data Integrity====

At the lowest level, ZFS uses checksums for every block of data that is written to disk. The checksum is checked whenever data is read to ensure that data has not been corrupted in some way. The idea is that if either the block or the checksum is corrupted, then recalculating the checksum for the block will result in a mismatch between the calculated and stored checksums. It is possible that both the block and checksum record could be corrupted, but the probability of the corruption being such that the corrupted block's checksum matches the corrupted checksum is exceptionally low.

In the event that a bad checksum is found, replication of data, in the form of "Ditto Blocks" provide an opportunity for recovery. A block pointer in ZFS is actually capable of pointing to multiple blocks, each of which contains duplicate data. By default, duplicate blocks are only stored for file system metadata, but this can be expanded to user data blocks as well. When a bad checksum is read, ZFS is able to follow one of the other pointers in the block pointer to hopefully find a healthy block.

RAID setups are particularly well suited to ZFS, since there is already an abstraction between the physical storage and the zpools. Besides protecting from outright total disk failure, if a bad checksum is found, there is the possibility that one of the alternate disks has a healthy version. If these errors accumulate, it can signal an impending drive failure. When a drive does fail, some of our tolerance for data loss is consumed; that is, the system is operating at less than 100% redundancy (however that is defined for the system at hand). To address this, ZFS supports "hot spares", idle drives that can be brought online automatically when another drive fails so that full redundancy can be rebuilt with minimal delay, hopefully in time for the next drive failure.

With block-by-block data integrity well in hand, ZFS also employs a transactional update model to ensure that higher level data structures remain consistent. Rather than use a journal to allow for quick consistency checking in the event of a system crash, ZFS uses a copy-on-write model. New disk structures are written out in a detached state. Once these structures have been written and checked, then they are connected to the existing disk structures in one atomic write, with the structures they replace becoming disconnected.

At the user level, ZFS supports file-system snapshots. Essentially, a clone of the entire file system at a certain point in time is created. In the event of accidental file deletion, a user can access an older version out of a recent snapshot.

====Data Deduplication====

Data Deduplication is a method of interfile storage compression, based around the idea of storing any one block of unique data only once physically, and logically linking that block to each file that contains that data. Effective use of data deduplication can reduce the space and power requirements of physical storage, but only if you have data that lends itself to deduplication.

Data Deduplication schemes are typically implemented using hash-tables, and can be applied to whole files, sub files (blocks), or as a patch set. There is an inherit trade off between the granularity of your deduplication algorithm and the resources needed to implement it. In general, as you consider smaller blocks of data for deduplication, you increase your "fold factor", that is, the difference between the logical storage provided vs. the physical storage needed. At the same time, however, smaller blocks means more hash table overhead and more CPU time needed for deduplication and for reconstruction.

The actual analysis and deduplication of incoming files can occur in-band or out-of-band. In-band deduplication means that the file is analyzed as it arrives at the storage server, and written to disk in its already compressed state. While this method requires the least over all storage capacity, resource constraints of the server may limit the speed at which new data can be ingested. In particular, the server must have enough memory to store the entire deduplication hash table in memory for fast comparisons. With out-of-band deduplication, inbound files are written to disk without any analysis (so, in the traditional way). A background process analyzes these files at a later time to perform the compression. This method means higher overall disk I/O is needed, which can be a problem if the disk (or disk array) is already at I/O capacity.

In the case of ZFS, which is typically hosted as a server-side file system, the server itself performs all of the deduplication and reconstruction; the entire process is transparent to the client. ZFS assumes that it is running on a highly multi-threaded operating system and that CPU cycles are in greater abundance than disk I/O cycles, and thus performs the deduplication in-band.

== References ==

* Mandagere, N., Zhou, P., Smith, M. A., and Uttamchandani, S. 2008. [http://portal.acm.org.proxy.library.carleton.ca/citation.cfm?id=1462739 Demystifying data deduplication]. In Proceedings of the ACM/IFIP/USENIX Middleware '08 Conference Companion (Leuven, Belgium, December 01 - 05, 2008). Companion '08. ACM, New York, NY, 12-17.

* Geer, D.; , [http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4712493 "Reducing the Storage Burden via Data Deduplication,"] Computer , vol.41, no.12, pp.15-17, Dec. 2008

* Bonwick, J.; [http://blogs.sun.com/bonwick/en_US/entry/zfs_dedup ZFS Deduplication]. Jeff Bonwick's Blog. November 2, 2009.

* Andrew Li, Department of Computing Macquarie University, Zettabyte File System Autopsy: Digital Crime Scene Investigation for Zettabyte File System

* Zhang, Y.; [http://www.cs.wisc.edu/wind/Publications/zfs-corruption-fast10.pdf End-to-end Data Integrity for File Systems: A ZFS Case Study]. FAST'10: Proceedings of the 8th USENIX conference on File and storage technologies. USENIX Association, Berkley, CA, USA.

COMP 3000 Essay 1 2010 Question 9

2010-10-10T22:46:14Z

Gbint: added some real meat to the data integrity section

=Question=

What requirements distinguish the Zettabyte File System (ZFS) from traditional file systems? How are those requirements realized in ZFS, and how do other operating systems address those same requirements? (Please discuss legacy, current, and in-development systems.)

=Answer=

ZFS was developed by Sun Microsystems (now owned by Oracle) as a server class file systems. This differs from most file systems which were developed as desktop file systems that could be used by servers. With the server being the target for the file system particular attention was paid to data integrity, size and speed.

One of the most significant ways in which the ZFS differs from traditional file systems is the level of abstraction. While a traditional file system abstracts away the physical properties of the media upon which it lies i.e. hard disk, flash drive, CD-ROM, etc. ZFS abstracts away if the file system lives one or many different pieces of hardware or media. Examples include a single hard drive, an array of hardrives, a number of hard drives on non co-located systems.

One of the mechanisms that allows this abstraction is that the volume manager which is normally a program separate from the file system in traditional file systems is moved into ZFS.

ZFS is a 128-bit file system allowing this allows addressing of 2128 bytes of storage.

=== Major Features of ZFS ===

====Physical Layer Abstraction====

* volume management and file system all in one
* file systems on top of zpools on top of vdevs on top of physical devices
* file systems easily and often span over many physical devices.
* ridiculous capacity

====Data Integrity====

At the lowest level, ZFS uses checksums for every block of data that is written to disk. The checksum is checked whenever data is read to ensure that data has not been corrupted in some way. The idea is that if either the block or the checksum is corrupted, then recalculating the checksum for the block will result in a mismatch between the calculated and stored checksums. It is possible that both the block and checksum record could be corrupted, but the probability of the corruption being such that the corrupted block's checksum matches the corrupted checksum is exceptionally low.

In the event that a bad checksum is found, replication of data, in the form of "Ditto Blocks" provide an opportunity for recovery. A block pointer in ZFS is actually capable of pointing to multiple blocks, each of which contains duplicate data. By default, duplicate blocks are only stored for file system metadata, but this can be expanded to user data blocks as well. When a bad checksum is read, ZFS is able to follow one of the other pointers in the block pointer to hopefully find a healthy block.

RAID setups are particularly well suited to ZFS, since there is already an abstraction between the physical storage and the zpools. Besides protecting from outright total disk failure, if a bad checksum is found, there is the possibility that one of the alternate disks has a healthy version. If these errors accumulate, it can signal an impending drive failure. When a drive does fail, some of our tolerance for data loss is consumed; that is, the system is operating at less than 100% redundancy (however that is defined for the system at hand). To address this, ZFS supports "hot spares", idle drives that can be brought online automatically when another drive fails so that full redundancy can be rebuilt with minimal delay, hopefully in time for the next drive failure.

With block-by-block data integrity well in hand, ZFS also employs a transactional update model to ensure that higher level data structures remain consistent. Rather than use a journal to allow for quick consistency checking in the event of a system crash, ZFS uses a copy-on-write model. New disk structures are written out in a detached state. Once these structures have been written and checked, then they are connected to the existing disk structures in one atomic write, with the structures they replace becoming disconnected.

At the user level, ZFS supports file-system snapshots. Essentially, a clone of the entire file system at a certain point in time is created. In the event of accidental file deletion, a user can access an older version out of a recent snapshot.

====Data Deduplication====

Data Deduplication is a method of interfile storage compression, based around the idea of storing any one block of unique data only once physically, and logically linking that block to each file that contains that data. Effective use of data deduplication can reduce the space and power requirements of physical storage, but only if you have data that lends itself to deduplication.

Data Deduplication schemes are typically implemented using hash-tables, and can be applied to whole files, sub files (blocks), or as a patch set. There is an inherit trade off between the granularity of your deduplication algorithm and the resources needed to implement it. In general, as you consider smaller blocks of data for deduplication, you increase your "fold factor", that is, the difference between the logical storage provided vs. the physical storage needed. At the same time, however, smaller blocks means more hash table overhead and more CPU time needed for deduplication and for reconstruction.

The actual analysis and deduplication of incoming files can occur in-band or out-of-band. In-band deduplication means that the file is analyzed as it arrives at the storage server, and written to disk in its already compressed state. While this method requires the least over all storage capacity, resource constraints of the server may limit the speed at which new data can be ingested. In particular, the server must have enough memory to store the entire deduplication hash table in memory for fast comparisons. With out-of-band deduplication, inbound files are written to disk without any analysis (so, in the traditional way). A background process analyzes these files at a later time to perform the compression. This method means higher overall disk I/O is needed, which can be a problem if the disk (or disk array) is already at I/O capacity.

In the case of ZFS, which is typically hosted as a server-side file system, the server itself performs all of the deduplication and reconstruction; the entire process is transparent to the client. ZFS assumes that it is running on a highly multi-threaded operating system and that CPU cycles are in greater abundance than disk I/O cycles, and thus performs the deduplication in-band.

== References ==

* Mandagere, N., Zhou, P., Smith, M. A., and Uttamchandani, S. 2008. [http://portal.acm.org.proxy.library.carleton.ca/citation.cfm?id=1462739 Demystifying data deduplication]. In Proceedings of the ACM/IFIP/USENIX Middleware '08 Conference Companion (Leuven, Belgium, December 01 - 05, 2008). Companion '08. ACM, New York, NY, 12-17.
* Geer, D.; , [http://ieeexplore.ieee.org.proxy.library.carleton.ca/xpls/abs_all.jsp?arnumber=4712493 "Reducing the Storage Burden via Data Deduplication,"] Computer , vol.41, no.12, pp.15-17, Dec. 2008
* Bonwick, J.; [http://blogs.sun.com/bonwick/en_US/entry/zfs_dedup ZFS Deduplication]. Jeff Bonwick's Blog. November 2, 2009.

* Andrew Li, Department of Computing Macquarie University, Zettabyte File System Autopsy: Digital Crime Scene Investigation for Zettabyte File System

Talk:COMP 3000 Essay 1 2010 Question 9

2010-10-09T16:29:49Z

Gbint: comments

== Essay Format ==

I started working on the main page. The bullets are to be expanded. Other group are are working in their respective discussion pages but I think it's all right to put our work in progress on the front page. Thoughts?--[[User:Lmundt|Lmundt]] 16:14, 6 October 2010 (UTC)
* [[User:Gbint|Gbint]] 02:03, 7 October 2010 (UTC) Lmundt; what do you think of listing the capacities of the file system under major features? I was thinking that we could overview the features in brief, then delve into each one individually.
* --[[User:Lmundt|Lmundt]] 14:31, 7 October 2010 (UTC) I was thinking about the major structure... I like what your suggesting in one section. So here is the structure I am thinking of.

* Intro
* Section One ZFS
** Major feature 1
** Major feature 2
** Major feature 3
* Section Two Legacy File Systems
** Legacy File System1( FAT32 ) - what it does
** Legacy File System2( ext2 ) - what it does
** Contrast them with ZFS
* Section Three Current File Systems
** NTFS?
** ext4?
** Contrast them with ZFS
* Section Four future file Systems
** BTRFS
** WinFS or ??
** Contrast them with ZFS
* Conclusion

What does everyone think of this format? While everyone should contribute to section one we could divvy up the rest.

[[User:Gbint|Gbint]] 16:29, 9 October 2010 (UTC) The layout looks good; I filled out the data dedup section. I think it has reasonable coverage while staying away from becoming it's own essay just on deduplication.

The legacy file systems are really not even in the same world as ZFS, so I think the contrasting section should cover a lot of how storage needs have changed.

The current file systems are capable of being expanded into large pools of storage with good redundancy and even advanced features like data deduplication, but they are only a component in a chain of tools (like ext4 + lvm + mdraid + opendedup) rather than an full end-to-end solution.

== Sources ==

Not from your group. Found a file which goes to the heart of your problem
[http://www.oracle.com/technetwork/server-storage/solaris/overview/zfs-14990
2.pdf ZFSDatasheet]
[[User:Gautam|Gautam]] 22:50, 5 October 2010 (UTC)

Thanks will take a look at that.--[[User:Lmundt|Lmundt]] 16:12, 6 October 2010 (UTC)

[[User:Gbint|Gbint]] 01:45, 7 October 2010 (UTC) paper from Sun engineers explaining why they came to build ZFS, the problems they wanted to solve:
* PDF: http://www.timwort.org/classp/200_HTML/docs/zfs_wp.pdf
* HTML: http://74.125.155.132/scholar?q=cache:6Ex3KbFo4lYJ:scholar.google.com/+zettabyte+file+system&hl=en&as_sdt=2000

Excellent article.[[User:Lmundt|Lmundt]] 14:24, 7 October 2010 (UTC)

Not too exciting but it looks like an easy read http://arstechnica.com/hardware/news/2008/03/past-present-future-file-systems.ars [[User:Lmundt|Lmundt]] 14:40, 7 October 2010 (UTC)

the [http://en.wikipedia.org/wiki/Comparison_of_file_systems wikipedia comparison] has some good tables, and if you click the various categories you can learn quite a bit about the various important features //not your group. [[User:Rift|Rift]] 18:56, 7 October 2010 (UTC)

Hey, I'm not from your group but I found this slideshow that was really handy in the assignment! http://www.slideshare.net/Clogeny/zfs-the-last-word-in-filesystems - nshires

------

Hey there. I'm not a member of your group. But you guys might want to look at this Wiki-page from the SolarisInternals website. I used it today for our assignment, a lot of interesting and in-depth breakdown of the ZFS file system: http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide#ZFS_Performance_Considerations

-- Munther

COMP 3000 Essay 1 2010 Question 9

2010-10-09T16:22:05Z

Gbint: skeletal pla section

COMP 3000 Essay 1 2010 Question 9

2010-10-09T16:14:40Z

Gbint: added some real meat to the deduplication section

Talk:COMP 3000 Essay 1 2010 Question 9

2010-10-07T02:05:38Z

Gbint: reorg of discussion page

Talk:COMP 3000 Essay 1 2010 Question 9

2010-10-07T02:03:42Z

Gbint: /* Sources */

== Sources ==

Not from your group. Found a file which goes to the heart of your problem
[http://www.oracle.com/technetwork/server-storage/solaris/overview/zfs-149902.pdf ZFSDatasheet]
[[User:Gautam|Gautam]] 22:50, 5 October 2010 (UTC)

Thanks will take a look at that.--[[User:Lmundt|Lmundt]] 16:12, 6 October 2010 (UTC)

I started working on the main page. The bullets are to be expanded. Other group are are working in their respective discussion pages but I think it's all right to put our work in progress on the front page. Thoughts?--[[User:Lmundt|Lmundt]] 16:14, 6 October 2010 (UTC)
* [[User:Gbint|Gbint]] 02:03, 7 October 2010 (UTC) Lmundt; what do you think of listing the capacities of the file system under major features? I was thinking that we could overview the features in brief, then delve into each one individually.

[[User:Gbint|Gbint]] 01:45, 7 October 2010 (UTC) paper from Sun engineers explaining why they came to build ZFS, the problems they wanted to solve:
* PDF: http://www.timwort.org/classp/200_HTML/docs/zfs_wp.pdf
* HTML: http://74.125.155.132/scholar?q=cache:6Ex3KbFo4lYJ:scholar.google.com/+zettabyte+file+system&hl=en&as_sdt=2000

COMP 3000 Essay 1 2010 Question 9

2010-10-07T02:01:08Z

Gbint: added data deduplication to the overview of ZFS features; minor editing of existing text

Talk:COMP 3000 Essay 1 2010 Question 9

2010-10-07T01:45:29Z

Gbint: Added source

== Sources ==

Not from your group. Found a file which goes to the heart of your problem
[http://www.oracle.com/technetwork/server-storage/solaris/overview/zfs-149902.pdf ZFSDatasheet]
[[User:Gautam|Gautam]] 22:50, 5 October 2010 (UTC)

Thanks will take a look at that.--[[User:Lmundt|Lmundt]] 16:12, 6 October 2010 (UTC)

I started working on the main page. The bullets are to be expanded. Other group are are working in their respective discussion pages but I think it's all right to put our work in progress on the front page. Thoughts?--[[User:Lmundt|Lmundt]] 16:14, 6 October 2010 (UTC)

[[User:Gbint|Gbint]] 01:45, 7 October 2010 (UTC) paper from Sun engineers explaining why they came to build ZFS, the problems they wanted to solve:
* PDF: http://www.timwort.org/classp/200_HTML/docs/zfs_wp.pdf
* HTML: http://74.125.155.132/scholar?q=cache:6Ex3KbFo4lYJ:scholar.google.com/+zettabyte+file+system&hl=en&as_sdt=2000

Talk:COMP 3000 Essay 1 2010 Question 7

2010-10-05T19:50:31Z

Gbint: Added a link to a paper

== Group 7 ==

Let us start out by listing down our names and email id (preffered).

Gautam Akiwate <gautam.akiwate@gmail.com>

== Sources ==

[[User:Gbint|Gbint]] 19:50, 5 October 2010 (UTC) Not in this group, but I thought that this paper was excellent: http://www.sandia.gov/~rcmurph/doc/qt_paper.pdf