Soma-notes - User contributions [en]

Talk:COMP 3000 Essay 2 2010 Question 2

2010-12-03T05:57:37Z

Gsmith6: /* Comments & Discussion */

=Comments & Discussion=

alrighty then.... I'm done for the night, I've just gone through it and it seems to flow nicely, and I made a few minor corrections here and there. I did have to make a major edit where I wrote all of the security invariants out and failed to notice that Youcef had done a similar thing just below it. Oh well.... It's fixed now. Night everyone
--[[User:Gsmith6|Gsmith6]] 05:57, 3 December 2010 (UTC)

I've added a few paragraphs in the background and the contribution sections including their references. Feel free to edit or add to them
--[[User:Ymoussou|Youcef M.]] 04:05, 3 December 2010 (UTC)

Don't forget your references guys!
--[[User:Ymoussou|Youcef M.]] 00:30, 3 December 2010 (UTC)

I'm adding bits and pieces here and there in most of the sections including references.

--[[User:Ymoussou|Youcef M.]] 22:01, 2 December 2010 (UTC)

--Writing up a critique of the researches evaluation methods for the critique section now

[[User:Atubman|Atubman]] 21:25, 2 December 2010 (UTC)

I've added to the Contribution part of the essay. I've basically explained as much as I thought was pertinent in what the section was asking for but don't be shy to add more!

--[[User:Selliot3|Selliot3]] 14:11, 2 December 2010 (UTC)

IBOS is designed to talk directly to the hardware. That's why they have everything at the lower level. All that extra TCB in other browsers was for all the extra stuff like services, OS components blah blah. I get their drift and how less code is secure but I don't get how they did it! From where do they get the services they claim to have taken out but still operates the same as any other browser. There is a catch somewhere but I can't find it in the paper or maybe I'm blind. I've been reading a lot of text but I got nowhere, its either too complex or not close to what I'm looking for.

--[[User:Ymoussou|Youcef M.]] 04:19, 2 December 2010 (UTC)

Brief description of the research problem below. (Still Needs expanding/fleshing out. can anyone help expand on why exactly shrinking the TCB will be more secure. I'm fuzzy on that)

The IBOS attempts to improve the security of web browsers. The writers argue that the large size of the trusted code bases (TCB) which modern web browsers make use of increases the possibility of a security hole. For example a hijacked window manager could be used to draw a fake phishing website overtop a web browser. The researchers solution is drastically shrinking the size of the TCB. The TCB is shrunk by turning the web browser into an operating system in itself with direct access to hardware abstractions.

Meanwhile, the IBOS must still support existing web applications while maintaining security.

[[User:Atubman|Atubman]] 03:36, 2 December 2010 (UTC)

This is what I understand of the TCB. Basically it is all of the components that are essential to the security of the computer system. Pretty much the stuff inside the kernel that if is compromised is "bad news bears". By removing things like device drivers, you reduce the TCB quite considerably (like probably 10s of thousands of lines of code). The smaller the TCB, the less of a chance that you have of an essential component getting corrupted. I don't know if I'm explaining it right, but maybe someone else can expand on what I've written here.
--[[User:Gsmith6|Gsmith6]] 16:13, 2 December 2010 (UTC)

EDIT: I've pretty much explained the background concept behind IBOS and I kind of added the way it's executed near the end. Feel free to move that into the research section.

I can work on the background of IBOS

--[[User:Selliot3|Selliot3]] 23:03, 22 November 2010 (UTC)

It seems we only have 5/7 members. We should start splitting up the tasks and assign who gets what. So if everybody writes what section they would like to work on that would be great.

--[[User:Ymoussou|Youcef M.]] 15:19, 20 November 2010 (UTC)

I'll do the contribution section. I'll be reading through the paper thoroughly today and taking notes as I go. I'll post them later on this page as a sort of cheat-sheet/reminder. --[[User:Gsmith6|Gsmith6]] 17:45, 25 November 2010 (UTC)

=Group Members=

Leave your name and e-mail address if you are assigned to this question.

[[User:Ymoussou|Youcef M.]] moussoud@gmail.com

I am alive and still in the class, selliot3@connect.carleton.ca

--[[User:Selliot3|Selliot3]] 18:12, 15 November 2010 (UTC)

Still in the class, andrewtubman84@gmail.com

[[User:Atubman|Atubman]]

I'm here. I have received an email reply from John Vanden Heuvel as well (he may not see this) gsmith0413@gmail.com
--[[User:Gsmith6|Gsmith6]] 22:31, 15 November 2010 (UTC)

[[User:vviveka2|vG]]

I am here... and replied to the email

=Question 2 members=

Elliott Charles selliot3

Moussoud Youcef ymoussou

Pharand Alexandre apharan2

Smith Geoffrey gsmith6

Tubman Andrew atubman

Vanden Heuvel John jvheuvel

Vivekanandarajah Vijitharan vviveka2

=Raw Information=

The web itself is ubiquitous which a person can use for communication; banking, business, social networking and it can be useful for other purposes. There are different type of vulnerabilities web applications, browser, OS and library vulnerabilities. Insecure web browsers are monolithic, and they are easy to exploit. Secure web browser such as chrome isolate web applications and it still contain huge trusted computing base (TCB). Browser abstractions as the first-class OS, contains reduced TCB for web browser and it also have protection to withstand attacks to most components. [[User:vviveka2|vG]]

=Extra Resources=
http://www.cs.uiuc.edu/homes/stang6/ibos.html#slide1
I found some presentation slides by Shuo Tang, Haohui Mai and Sam King, the authors and developers of IBOS
--[[User:Gsmith6|Gsmith6]] 22:35, 25 November 2010 (UTC)

COMP 3000 Essay 2 2010 Question 2

2010-12-03T05:53:53Z

Gsmith6:

==Paper==
'''Trust and Protection in the Illinois Browser Operating System'''

http://www.usenix.org/events/osdi10/tech/full_papers/Tang.pdf

Shuo Tang, Haohui Mai, Samuel T. King

''University of Illinois at Urbana-Champaig''

Presentation slides to go along with the paper: Trust and Protection in the Illinois Browser Operating System. http://www.cs.uiuc.edu/homes/stang6/ibos.html#slide1

==Background Concepts==
In the world we are in, the web is everywhere and runs in different operating systems and browsers. The Illinois Browser Operating System (IBOS) is not just a new browser to improve security, it is also a full operating system. It was developed by three graduate students at the University of Illinois. It’s main goal is to expose browser-level abstractions at the lowest possible software layer, reducing the trusted computing base for web browsers. Many websites and web applications have become major targets for attackers and hackers. These attackers are always finding new ways of exploiting even the most secure systems. Just recently, cross-site scripting (XSS) has become the most common security vulnerability over the age old buffer overflow [[#References | [1]]].

Plenty of research has gone in to improving security among the various web browsers on the market today but all browsers still remain susceptible to attacks on the lower layers. Compromised Ethernet drivers can send sensitive HTTP packets to third parties, compromised storage modules can send persistent data to unwanted viewers and compromised window managers can overlay fake interfaces common in phishing attacks [[#References | [1]]]. Common web browsers run on top of commodity operating systems with shared system services and user-mode libraries, increasing the trusted computing base(TCB). IBOS looks to solve this issue by exposing browser-level abstractions rather than just general-purpose abstractions. Important concepts such as cookies, HTTP connections and tabs for displaying pages are all brought into the browser abstraction layer. By moving all of these subsystems, the IBOS system ultimately aims to reduce the computer's TCB.

===Monolithic vs Modular===

The internet today has become an important part of our lives, it is used everywhere and in everyday life. It all started with the user only being allowed to read and browse the content, but today the expansion has led to allow the ability to write our own content and have our own impact on the rest of the world. The introduction of Web 2.0 brought upon an enormous change to the web, as well as introducing the concept of security concerns. Modern web applications are mostly designed to meet a newer modular architecture.

Modularly designed architecture browsers such as Chrome are designed in a way that each browsing instance is assigned to a unique operating system process. It is designed to perform much better in terms of fault-tolerance, accountability, security, memory management and performance. Chrome has the ability to be able to run even when another web program crashes. Memory management in a modular architecture is designed to handle each process at a time and when the program closes the memory space is ready to be used by another program. Scheduling for modular browsers handled at the OS level and web programs are able to run parallel.

Monolithic architectures browser such as Firefox, are monolithic and easy to exploit, since they run in a single address space. It is designed in a way that all the components for the web run in a single process. Disadvantages about monolithic architectures are that if a single web program crashes, then all of the other web browsers also crash. This can cause all of the data to be lost that is currently saved. Memory management is also considered poor in a monolithic architecture because it allocates memory at the beginning of the web program and may contain leaks. This is a major problem, because when the program finishes execution, the memory leak will have grown exponentially.

===TCB===

The TCB is the hardware and software that is critical to the computer's security. It's a combination of kernel and trusted processes made of hardware, firmware, and software that are critical to a computer's security [[#References | [2]]]. In the words of Lampson et al., TCB is defined as:

''"a small amount of software and hardware that security depends on and that we distinguish from a much larger amount that can misbehave without affecting security."'' [[#References | [3]]].

Modern operating system-browser combinations have massive TCBs that may have several millions of lines of code. By extracting components such as device drivers from the kernel, one can lower a systems TCB considerably. If a device driver is outside of the TCB and becomes corrupted, the effects would not be too severe, but if the driver is left in the TCB, then the results could be cataclysmic. By removing elements from the TCB, it becomes smaller, thereby reducing the risk of having an attack get inside.

==Research Problem==
Modern browsers, such as Google Chrome and Mozilla Firefox, are continuously being revised and updated to keep ahead of the latest attacks, but continuously have hundreds of security vulnerabilities. Most of these attacks are simple, mostly harmless assaults on web applications, but many attacks are on the browser or even the operating system and its libraries. Since the browser runs lower on the shared storage stack, a successful attack on a browser can have horrible repercussions because it gives access to all of the browser data for all of the web applications. It also provides the attacker with access to other resources on the system which is being exploited. An attack on the operating system can be disastrous if it is successful and may cause serious damage to the entire system. This is partly because the attackers can access arbitrary states and events, allowing them to have full control over the system. The focus of this research is to prevent and decrease the attacks on the browser, libraries, operating systems and system services.

==Contribution==

===Architecture and Design===
The authors have developed IBOS to reduce security risks, without compromising speed and efficiency. One of the ways they have achieved this is through their implementation of process creation. Essentially there are two types of processes. A web page instance and a traditional process. Any time the user opens a new tab, clicks on a link, or enters a web address in the uniform resource locator(URL) bar, the IBOS kernel creates a new process. Upon creating a web page instance process, the kernel labels it with the originating address of the HTTP request. If a web site such as ''facebook.com'' decides to host an outside script, also known as an iframe, from another website, the kernel creates a new process for the embedded script and labels it appropriately. Traditional processes are every other process that is created for the local machine. These processes are simply labeled as ''localhost''.

An IBOS label contains a protocol, domain and port. By creating unique labels for each web page instance, the kernel can isolate them from one another. This prevents a compromised component from taking control of other processes. Also by labeling where requests come from, the IBOS kernel can ensure that the data it is receiving is in fact from the expected origin. Each time a process makes a request, the kernel checks to ensure that the process is in fact allowed to access the requested resources.

Each time an HTTP request comes in over the network, the network process renders the HTTP into a TCP stream, which is in turn converted into a series of Ethernet frames that are sent to the network interface controller(NIC). By doing all of this, the process is excluded from accessing any data structure. The process must first pass through the kernel before it receives what it has requested.

Through the use of twelve security invariants, the development team ensures that subsystems behave according to their intended purpose without interfering with other subsystems. The invariants are designed for different components such as driver, storage, network processes, and UI.

For driver invariants, IBOS ensures drivers do not access DMA buffer directly and devices can only access validated DMA buffers. their approach is to separate the the management of device control registers from the use of device buffers. Using this separation, processes can fill device specific buffers for DMA transfers, and the IBOS kernel instructs the driver to enforce the device to use a verified DMA buffer.

In the storage invariant, IBOS encrypts all objects before passing them to the storage subsystem. That way all of the key-value pairs maintain confidentiality and integrity even if the storage stack itself becomes compromised.

The network process invariants are where the security is focused. The first two invariants are: The kernel must route network requests from web page instances to the proper network process, and the kernel must route Ethernet frames from the NIC to the proper network processes. To enforce all of these invariants, IBOS puts all network processes in their own protection domains. By putting network processes in their own protection domains, the kernel naturally ensures that network requests from web page instances and Ethernet frames from the NIC are routed to the correct network process. The third invariant is that the Ethernet frames from network processes to the NIC must have an IP address and TCP port that matches the origin of the network process. IBOS kernel checks all outgoing Ethernet frames before sending them to the NIC to check the IP address and TCP port against the label of the sending network process. The fourth invariant is HTTP data from network processes to web page instances must adhere to the SOP. The IBOS kernel inspects HTTP data before forwarding it to the appropriate web page instance and drops any HTML documents from different origins. The last invariant is that network processes for different web page instances must remain isolated by labelling the origin of the web page instance.

As for the UI subsystem, IBOS kernel enforces 3 invariants. The first is the browser and web page content displays are isolated by using a frame buffer video driver and page protections to isolate portions of the screen. The second invariant is that only the current tab can access the screen, mouse, and keyboard by mapping the frame buffer memory region and routing mouse and keyboard events into the currently active web page instance. the final invariant is the URL of the current tab is displayed to the user by using the kernel display area to display the URL.

The IBOS kernel also offers custom security by allowing web sites to specify a server-side policy file that IBOS retrieves to restrict network accesses for a web page instance.

The security of the window manager in the browser is achieved by dividing the window into three exclusively owned horizontal sections. The top bar of the page can only be accessed by the kernel. This is where the URL, labelled by the kernel is displayed. The second section is controlled by the browser UI, which the developers have called the chrome. This is where the back buttons, URL bar and other objects found in various browsers are located. The rest of the page is reserved for the web page instance. This is where the web page is displayed. Although this division of responsibilities cannot completely eradicate phishing attacks, it does however give the user a better chance to realize that something is wrong. By displaying the URL as defined by the kernel in the uppermost bar, the user can see exactly which website is being displayed.

IBOS has a considerably smaller TCB compared to other modern browsers. Where both Chrome and Firefox come in at over 4 million plus lines of code in their trusted computing base, IBOS has only about 42,000. Since IBOS isolates each process, it was also able to prevent between 75-100% of vulnerabilities from affected components on a machine. Using Chrome, the researchers tested 175 known issues on the IBOS kernel which ranged from memory exploits to interface spoofing. Out of all the known issues, IBOS was able to prevent 135 or 77% of the issues whereas Chrome was only able to contain 83 of them. The issue is that Chrome is able to catch exploits in its rendering engine since it is in a sandbox but any exploits that took advantage of the browser kernel could not be prevented. This is not a problem for IBOS because many of the browser components inside the trusted computing base in Chrome have been brought outside of the IBOS TCB limiting what can be done with exploitation. IBOS also contains tabs abstraction which multiplexes the display between different web pages instances. Input devices are programmed to route to a visible tab, which prevents keylogger from hijacking another instance's session.

===Performance===
In terms of performance, IBOS is comparable to the two best performing web browsers currently released: Firefox and Chrome. For websites such as Google Maps and Facebook, IBOS actually performs much better than Firefox while loading pages. This may be due partly to the fact that IBOS was developed with the WebKit engine, which has been optimized to run Google Maps. For Facebook and Wikipedia, sites that use many HTTP requests, IBOS performs slightly slower than the other two browsers, but for the others, where there are only a few HTTP requests, IBOS runs just as quickly as the others.

==Critique==

===Structure===
This paper was very well organized and well executed. It naturally flows and keeps order in what it is trying to explain without the need to flip back and reference another piece of content in the paper. Starting with the core mechanics of why it is needed to how the kernel is organized and working its way up to many high-level pieces of information it felt like a natural progression of ideas, giving you the information you need to understand upcoming concepts.

===Evaluation===
The evaluation of the IBOS security has some flaws, it is not very thorough and the data is potentially confounding. When the authors were testing two systems that could not be compared head-to-head, they made many assumptions that could possibly be wrong.

The IBOS has shown through internal testing that it is able to resist 77% of attacks from a set of 175 security bugs whereas Chrome is only able prevent 46%. The improvement sounds impressive however, the set of security bugs they tested against was obtained from Google “Chrome’s bug tracker”. The fact they are comparing known security flaws in Chrome against the new IBOS makes their improvement of 31% far less impressive.

In addition, Their initial test set contained 217 bugs with duplicates removed, and 42 bugs were omitted because they were denial of service attacks and the IBOS does not protect against that form of attack. It is understandable this is out of the scope of this research. However, that is a big set of flaws which are not addressed.

Furthermore, the researchers only compared their results against the Chrome web browser. A comparison which also includes other browsers such as Mozilla Firefox, Internet Explorer and Safari would be much more compelling.

==References==

[1] CVE - Common Vulnerabilities and Exposures (CVE). http://cve.mitre.org.

[2] Rushby, John (1981). "Design and Verification of Secure Systems". 8th ACM Symposium on Operating System Principles. Pacific Grove, California, US. pp. 12–21.

[3] B. Lampson, M. Abadi, M. Burrows and E. Wobber, Authentication in Distributed Systems: Theory and Practice, ACM Transactions on Computer Systems 1992, on page 6.

COMP 3000 Essay 2 2010 Question 2

2010-12-03T05:26:23Z

Gsmith6: /* Architecture and Design */

==Paper==
'''Trust and Protection in the Illinois Browser Operating System'''

http://www.usenix.org/events/osdi10/tech/full_papers/Tang.pdf

Shuo Tang, Haohui Mai, Samuel T. King

''University of Illinois at Urbana-Champaig''

Presentation slides to go along with the paper: Trust and Protection in the Illinois Browser Operating System. http://www.cs.uiuc.edu/homes/stang6/ibos.html#slide1

==Background Concepts==
In the world we are in, the web is everywhere and runs in different operating systems and browsers. The Illinois Browser Operating System (IBOS) is not just a new browser to improve security, it is also a full operating system. It was developed by three graduate students at the University of Illinois. It’s main goal is to expose browser-level abstractions at the lowest possible software layer, reducing the trusted computing base for web browsers. Many websites and web applications have become major targets for attackers and hackers. These attackers are always finding new ways of exploiting even the most secure systems. Just recently, cross-site scripting (XSS) has become the most common security vulnerability over the age old buffer overflow [[#References | [1]]].

Plenty of research has gone in to improving security among the various web browsers on the market today but all browsers still remain susceptible to attacks on the lower layers. Compromised Ethernet drivers can send sensitive HTTP packets to third parties, compromised storage modules can send persistent data to unwanted viewers and compromised window managers can overlay fake interfaces common in phishing attacks [[#References | [1]]]. Common web browsers run on top of commodity operating systems with shared system services and user-mode libraries, increasing the trusted computing base(TCB). IBOS looks to solve this issue by exposing browser-level abstractions rather than just general-purpose abstractions. Important concepts such as cookies, HTTP connections and tabs for displaying pages are all brought into the browser abstraction layer. By moving all of these subsystems, the IBOS system ultimately aims to reduce the computer's TCB.

===Monolithic vs Modular===

The internet today has become an important part of our lives, it is used everywhere and in everyday life. It all started with the user only being allowed to read and browse the content, but today the expansion has led to allow the ability to write our own content and have our own impact on the rest of the world. The introduction of Web 2.0 brought upon an enormous change to the web, as well as introducing the concept of security concerns. Modern web applications are mostly designed to meet a newer modular architecture.

Modularly designed architecture browsers such as Chrome are designed in a way that each browsing instance is assigned to a unique operating system process. It is designed to perform much better in terms of fault-tolerance, accountability, security, memory management and performance. Chrome has the ability to be able to run even when another web program crashes. Memory management in a modular architecture is designed to handle each process at a time and when the program closes the memory space is ready to be used by another program. Scheduling for modular browsers handled at the OS level and web programs are able to run parallel.

Monolithic architectures browser such as Firefox, are monolithic and easy to exploit, since they run in a single address space. It is designed in a way that all the components for the web run in a single process. Disadvantages about monolithic architectures are that if a single web program crashes, then all of the other web browsers also crash. This can cause all of the data to be lost that is currently saved. Memory management is also considered poor in a monolithic architecture because it allocates memory at the beginning of the web program and may contain leaks. This is a major problem, because when the program finishes execution, the memory leak will have grown exponentially.

===TCB===

The TCB is the hardware and software that is critical to the computer's security. It's a combination of kernel and trusted processes made of hardware, firmware, and software that are critical to a computer's security [[#References | [2]]]. In the words of Lampson et al., TCB is defined as:

''"a small amount of software and hardware that security depends on and that we distinguish from a much larger amount that can misbehave without affecting security."'' [[#References | [3]]].

Modern operating system-browser combinations have massive TCBs that may have several millions of lines of code. By extracting components such as device drivers from the kernel, one can lower a systems TCB considerably. If a device driver is outside of the TCB and becomes corrupted, the effects would not be too severe, but if the driver is left in the TCB, then the results could be cataclysmic. By removing elements from the TCB, it becomes smaller, thereby reducing the risk of having an attack get inside.

==Research Problem==
Modern browsers, such as Google Chrome and Mozilla Firefox, are continuously being revised and updated to keep ahead of the latest attacks, but continuously have hundreds of security vulnerabilities. Most of these attacks are simple, mostly harmless assaults on web applications, but many attacks are on the browser or even the operating system and its libraries. Since the browser runs lower on the shared storage stack, a successful attack on a browser can have horrible repercussions because it gives access to all of the browser data for all of the web applications. It also provides the attacker with access to other resources on the system which is being exploited. An attack on the operating system can be disastrous if it is successful and may cause serious damage to the entire system. This is partly because the attackers can access arbitrary states and events, allowing them to have full control over the system. The focus of this research is to prevent and decrease the attacks on the browser, libraries, operating systems and system services.

==Contribution==

===Architecture and Design===
The authors have developed IBOS to reduce security risks, without compromising speed and efficiency. One of the ways they have achieved this is through their implementation of process creation. Essentially there are two types of processes. A web page instance and a traditional process. Any time the user opens a new tab, clicks on a link, or enters a web address in the uniform resource locator(URL) bar, the IBOS kernel creates a new process. Upon creating a web page instance process, the kernel labels it with the originating address of the HTTP request. If a web site such as ''facebook.com'' decides to host an outside script, also known as an iframe, from another website, the kernel creates a new process for the embedded script and labels it appropriately. Traditional processes are every other process that is created for the local machine. These processes are simply labeled as ''localhost''.

An IBOS label contains a protocol, domain and port. By creating unique labels for each web page instance, the kernel can isolate them from one another. This prevents a compromised component from taking control of other processes. Also by labeling where requests come from, the IBOS kernel can ensure that the data it is receiving is in fact from the expected origin. Each time a process makes a request, the kernel checks to ensure that the process is in fact allowed to access the requested resources.

Each time an HTTP request comes in over the network, the network process renders the HTTP into a TCP stream, which is in turn converted into a series of Ethernet frames that are sent to the network interface controller(NIC). By doing all of this, the process is excluded from accessing any data structure. The process must first pass through the kernel before it receives what it has requested.

Through the use of twelve security invariants, the development team ensures that subsystems behave according to their intended purpose. The following are all twelve security invariants:

SI 0: All components can only perform their designated functions.

SI 1: Drivers cannot access DMA buffers directly.

SI 2: Devices can only access validated DMA buffers.

SI 3: All of the key-value pairs maintain confidentiality and integrity even if the storage stack becomes compromised

-this is acheived by encrypting objects before passing them to the storage subsystem

SI 4: The kernel must route network requests from web page instances to the proper network process

SI 5: The kernel must route Ethernet frames from the NIC to the proper netwok processes

SI 6: Ethernet frames from network processes to the NIC must have an IP address and TCP port that matches the origin of the network process

SI 7: HTTP data from network process to web page instances must adhere to the SOP

SI 8: Network processes for different web page instances must remain isolated

SI 9: The browser chrome and web page content displays are isolated

SI 10: Only the current tab can access the screen, mouse and keyboard

SI 11: The URL of the current tab is displayed to the user

The security of the window manager in the browser is achieved by dividing the window into three exclusively owned horizontal sections. The top bar of the page can only be accessed by the kernel. This is where the URL, labelled by the kernel is displayed. The second section is controlled by the browser UI, which the developers have called the chrome. This is where the back buttons, URL bar and other objects found in various browsers are located. The rest of the page is reserved for the web page instance. This is where the web page is displayed. Although this division of responsibilities cannot completely eradicate phishing attacks, it does however give the user a better chance to realize that something is wrong. By displaying the URL as defined by the kernel in the uppermost bar, the user can see exactly which website is being displayed.

IBOS has considerably smaller TCB compared to other modern browsers. Where both Chrome and Firefox come in at over 4 million plus lines of code in their trusted computing base, IBOS has only about 42,000. Since IBOS isolates each process, it was also able to prevent between 75-100% of vulnerabilities from affected components on a machine. Using Chrome, the researchers tested 175 known issues on the IBOS kernel which ranged from memory exploits to interface spoofing. Out of all the known issues, IBOS was able to prevent 135 or 77% of the issues whereas Chrome was only able to contain 83 of them. The issue is that Chrome is able to catch exploits in its rendering engine since it is in a sandbox but any exploits that took advantage of the browser kernel could not be prevented. This is not a problem for IBOS because many of the browser components inside the trusted computing base in Chrome have been brought outside of the IBOS TCB limiting what can be done with exploitation. IBOS also contains tabs abstraction which multiplexes the display between different web pages instances. Input devices are programmed to route to a visible tab, which prevents keylogger from hijacking another instance's session.

===Performance===
In terms of performance, IBOS is comparable to the two best performing web browsers currently released: Firefox and Chrome. For websites such as Google Maps and Facebook, IBOS actually performs much better than Firefox while loading pages. This may be due partly to the fact that IBOS was developed with the WebKit engine, which has been optimized to run Google Maps. For Facebook and Wikipedia, sites that use many HTTP requests, IBOS performs slightly slower than the other two browsers, but for the others, where there are only a few HTTP requests, IBOS runs just as quickly as the others.

IBOS have the same functionality as Firefox and Chrome which uses the standard Web kit but it gives better performance.

===Security===
In addition to labels, IBOS has other approaches at reaching their goals in security. Security variants are a way for IBOS to enforce their subsystems to perform their designated functions without interfering with other subsystems. To reach this goal, they designed invariants for different components such as driver, storage, network processes, and UI invariants.

For driver invariants, IBOS insures drivers do not access DMA buffer directly and devices can only access validated DMA buffers. their approach is to separate the the management of device control registers from the use of device buffers. Using this separation, processes can fill device specific buffers for DMA transfers, and the IBOS kernel instructs the driver to enforce the device to use a verified DMA buffer.

In storage invariant IBOS encrypts all objects before passing them to the storage subsystem. That way All of the key-value pairs maintain confidentiality and integrity even if the storage stack itself becomes compromised.

The network process invariants is where the security is focused. The first two invariants are: The kernel must route network requests from web page instances to the proper network process, and the kernel must route Ethernet frames from the NIC to the proper network processes. To enforce all of these invariants, IBOS puts all network processes in their own protection domains. By putting network processes in their own protection domains, the kernel naturally ensures that network requests from web page instances and Ethernet frames from the NIC are routed to the correct network process. The third invariant is that the Ethernet frames from network processes to the NIC must have an IP address and TCP port that matches the origin of the network process. IBOS kernel checks all outgoing Ethernet frames before sending them to the NIC to check the IP address and TCP port against the label of the sending network process. The fourth invariant is HTTP data from network processes to web page instances must adhere to the SOP. The IBOS kernel inspects HTTP
data before forwarding it to the appropriate web page instance and drops any HTML documents from different origins. The last invariant is that network processes for different web page instances must remain isolated by labelling the origin of the web page instance.

As for the UI subsystem, IBOS kernel enforces 3 invariants. The first is the browser and web page content displays are isolated by using a frame buffer video driver and page protections to isolate portions of the screen. The second invariant is that only the current tab can access the screen, mouse, and keyboard by mapping the frame buffer memory region and routing mouse and keyboard events into the currently active web page instance. the final invariant is the URL of the current tab is displayed to the user by using the kernel display area to display the URL.

The IBOS kernel also offers custom security by allowing web sites to specify a server-side policy file that IBOS retrieves to restrict network accesses for a web page instance.

==Critique==

===Structure===
This paper was very well organized and executed. It naturally flows and keeps order in what it is trying to explain without the need to flip back and reference another piece of content in the paper. Starting with the core mechanics of why it is needed to how the kernel is organized and working its way up to many high-level pieces of information it felt like a natural progression of ideas, giving you the information you need to understand upcoming concepts.

===Evaluation===
The evaluation of the IBOS security has some flaws,it is not very thorough and the data set the testing against is potentially confounding.

The IBOS has shown through internal testing that it is able to resist 77% of attacks from a set of 175 security bugs whereas Chrome is only able prevent 46%. The improvement sounds impressive however, the set of security bugs they tested against was obtained from Google “Chrome’s bug tracker”. The fact they are comparing known security flaws in Chrome against the new IBOS makes their improvement of 31% far less impressive.

In addition, Their initial test set contained 217 bugs with duplicates removed, and 42 bugs were omitted because they were denial of service attacks and the IBOS does not protect against that form of attack. It is understandable this is out of the scope of this research. However, that is a big set of flaws which are not addressed.

Furthermore, the researches only compared their results against the Chrome web browser. A comparison which also includes other browsers such as Mozilla Firefox, Internet Explorer and Safari would be much more compelling.

==References==

[1] CVE - Common Vulnerabilities and Exposures (CVE). http://cve.mitre.org.

[2] Rushby, John (1981). "Design and Verification of Secure Systems". 8th ACM Symposium on Operating System Principles. Pacific Grove, California, US. pp. 12–21.

[3] B. Lampson, M. Abadi, M. Burrows and E. Wobber, Authentication in Distributed Systems: Theory and Practice, ACM Transactions on Computer Systems 1992, on page 6.

COMP 3000 Essay 2 2010 Question 2

2010-12-03T05:13:19Z

Gsmith6: /* Architecture and Design */

==Paper==
'''Trust and Protection in the Illinois Browser Operating System'''

http://www.usenix.org/events/osdi10/tech/full_papers/Tang.pdf

Shuo Tang, Haohui Mai, Samuel T. King

''University of Illinois at Urbana-Champaig''

Presentation slides to go along with the paper: Trust and Protection in the Illinois Browser Operating System. http://www.cs.uiuc.edu/homes/stang6/ibos.html#slide1

==Background Concepts==
In the world we are in, the web is everywhere and runs in different operating systems and browsers. The Illinois Browser Operating System (IBOS) is not just a new browser to improve security, it is also a full operating system. It was developed by three graduate students at the University of Illinois. It’s main goal is to expose browser-level abstractions at the lowest possible software layer, reducing the trusted computing base for web browsers. Many websites and web applications have become major targets for attackers and hackers. These attackers are always finding new ways of exploiting even the most secure systems. Just recently, cross-site scripting (XSS) has become the most common security vulnerability over the age old buffer overflow [[#References | [1]]].

Plenty of research has gone in to improving security among the various web browsers on the market today but all browsers still remain susceptible to attacks on the lower layers. Compromised Ethernet drivers can send sensitive HTTP packets to third parties, compromised storage modules can send persistent data to unwanted viewers and compromised window managers can overlay fake interfaces common in phishing attacks [[#References | [1]]]. Common web browsers run on top of commodity operating systems with shared system services and user-mode libraries, increasing the trusted computing base(TCB). IBOS looks to solve this issue by exposing browser-level abstractions rather than just general-purpose abstractions. Important concepts such as cookies, HTTP connections and tabs for displaying pages are all brought into the browser abstraction layer. By moving all of these subsystems, the IBOS system ultimately aims to reduce the computer's TCB.

===Monolithic vs Modular===

The internet today has become an important part of our lives, it is used everywhere and in everyday life. It all started with the user only being allowed to read and browse the content, but today the expansion has led to allow the ability to write our own content and have our own impact on the rest of the world. The introduction of Web 2.0 brought upon an enormous change to the web, as well as introducing the concept of security concerns. Modern web applications are mostly designed to meet a newer modular architecture.

Modularly designed architecture browsers such as Chrome are designed in a way that each browsing instance is assigned to a unique operating system process. It is designed to perform much better in terms of fault-tolerance, accountability, security, memory management and performance. Chrome has the ability to be able to run even when another web program crashes. Memory management in a modular architecture is designed to handle each process at a time and when the program closes the memory space is ready to be used by another program. Scheduling for modular browsers handled at the OS level and web programs are able to run parallel.

Monolithic architectures browser such as Firefox, are monolithic and easy to exploit, since they run in a single address space. It is designed in a way that all the components for the web run in a single process. Disadvantages about monolithic architectures are that if a single web program crashes, then all of the other web browsers also crash. This can cause all of the data to be lost that is currently saved. Memory management is also considered poor in a monolithic architecture because it allocates memory at the beginning of the web program and may contain leaks. This is a major problem, because when the program finishes execution, the memory leak will have grown exponentially.

===TCB===

The TCB is the hardware and software that is critical to the computer's security. It's a combination of kernel and trusted processes made of hardware, firmware, and software that are critical to a computer's security [[#References | [2]]]. In the words of Lampson et al., TCB is defined as:

''"a small amount of software and hardware that security depends on and that we distinguish from a much larger amount that can misbehave without affecting security."'' [[#References | [3]]].

Modern operating system-browser combinations have massive TCBs that may have several millions of lines of code. By extracting components such as device drivers from the kernel, one can lower a systems TCB considerably. If a device driver is outside of the TCB and becomes corrupted, the effects would not be too severe, but if the driver is left in the TCB, then the results could be cataclysmic. By removing elements from the TCB, it becomes smaller, thereby reducing the risk of having an attack get inside.

==Research Problem==
Modern browsers, such as Google Chrome and Mozilla Firefox, are continuously being revised and updated to keep ahead of the latest attacks, but continuously have hundreds of security vulnerabilities. Most of these attacks are simple, mostly harmless assaults on web applications, but many attacks are on the browser or even the operating system and its libraries. Since the browser runs lower on the shared storage stack, a successful attack on a browser can have horrible repercussions because it gives access to all of the browser data for all of the web applications. It also provides the attacker with access to other resources on the system which is being exploited. An attack on the operating system can be disastrous if it is successful and may cause serious damage to the entire system. This is partly because the attackers can access arbitrary states and events, allowing them to have full control over the system. The focus of this research is to prevent and decrease the attacks on the browser, libraries, operating systems and system services.

==Contribution==

===Architecture and Design===
The authors have developed IBOS to reduce security risks, without compromising speed and efficiency. One of the ways they have achieved this is through their implementation of process creation. Essentially there are two types of processes. A web page instance and a traditional process. Any time the user opens a new tab, clicks on a link, or enters a web address in the uniform resource locator(URL) bar, the IBOS kernel creates a new process. Upon creating a web page instance process, the kernel labels it with the originating address of the HTTP request. If a web site such as ''facebook.com'' decides to host an outside script, also known as an iframe, from another website, the kernel creates a new process for the embedded script and labels it appropriately. Traditional processes are every other process that is created for the local machine. These processes are simply labeled as ''localhost''.

An IBOS label contains a protocol, domain and port. By creating unique labels for each web page instance, the kernel can isolate them from one another. This prevents a compromised component from taking control of other processes. Also by labeling where requests come from, the IBOS kernel can ensure that the data it is receiving is in fact from the expected origin. Each time a process makes a request, the kernel checks to ensure that the process is in fact allowed to access the requested resources.

Each time an HTTP request comes in over the network, the network process renders the HTTP into a TCP stream, which is in turn converted into a series of Ethernet frames that are sent to the network interface controller(NIC). By doing all of this, the process is excluded from accessing any data structure. The process must first pass through the kernel before it receives what it has requested.

Through the use of twelve security invariants, the development team ensures that subsystems behave according to their intended purpose. The following are all twelve security invariants:

SI 0: All components can only perform their designated functions.

SI 1: Drivers cannot access DMA buffers directly.

SI 2: Devices can only access validated DMA buffers.

SI 3: All of the key-value pairs maintain confidentiality and integrity even if the storage stack becomes compromised

-this is acheived by encrypting objects before passing them to the storage subsystem

SI 4: The kernel must route network requests from web page instances to the proper network process

SI 5: The kernel must route Ethernet frames from the NIC to the proper netwok processes

SI 6: Ethernet frames from network processes to the NIC must have an IP address and TCP port that matches the origin of the network process

SI 7: HTTP data from network process to web page instances must adhere to the SOP

SI 8: Network processes for different web page instances must remain isolated

SI 9: The browser chrome and web page content displays are isolated

SI 10: Only the current tab can access the screen, mouse and keyboard

SI 11: The URL of the current tab is displayed to the user

IBOS has considerably smaller TCB compared to other modern browsers. Where both Chrome and Firefox come in at over 4 million plus lines of code in their trusted computing base, IBOS has only about 42,000. Since IBOS isolates each process, it was also able to prevent between 75-100% of vulnerabilities from affected components on a machine. Using Chrome, the researchers tested 175 known issues on the IBOS kernel which ranged from memory exploits to interface spoofing. Out of all the known issues, IBOS was able to prevent 135 or 77% of the issues whereas Chrome was only able to contain 83 of them. The issue is that Chrome is able to catch exploits in its rendering engine since it is in a sandbox but any exploits that took advantage of the browser kernel could not be prevented. This is not a problem for IBOS because many of the browser components inside the trusted computing base in Chrome have been brought outside of the IBOS TCB limiting what can be done with exploitation. IBOS also contains tabs abstraction which multiplexes the display between different web pages instances. Input devices are programmed to route to a visible tab, which prevents keylogger from hijacking another instance's session.

===Performance===
In terms of performance, IBOS is comparable to the two best performing web browsers currently released: Firefox and Chrome. For websites such as Google Maps and Facebook, IBOS actually performs much better than Firefox while loading pages. This may be due partly to the fact that IBOS was developed with the WebKit engine, which has been optimized to run Google Maps. For Facebook and Wikipedia, sites that use many HTTP requests, IBOS performs slightly slower than the other two browsers, but for the others, where there are only a few HTTP requests, IBOS runs just as quickly as the others.

IBOS have the same functionality as Firefox and Chrome which uses the standard Web kit but it gives better performance.

===Security===
In addition to labels, IBOS has other approaches at reaching their goals in security. Security variants are a way for IBOS to enforce their subsystems to perform their designated functions without interfering with other subsystems. To reach this goal, they designed invariants for different components such as driver, storage, network processes, and UI invariants.

For driver invariants, IBOS insures drivers do not access DMA buffer directly and devices can only access validated DMA buffers. their approach is to separate the the management of device control registers from the use of device buffers. Using this separation, processes can fill device specific buffers for DMA transfers, and the IBOS kernel instructs the driver to enforce the device to use a verified DMA buffer.

In storage invariant IBOS encrypts all objects before passing them to the storage subsystem. That way All of the key-value pairs maintain confidentiality and integrity even if the storage stack itself becomes compromised.

The network process invariants is where the security is focused. The first two invariants are: The kernel must route network requests from web page instances to the proper network process, and the kernel must route Ethernet frames from the NIC to the proper network processes. To enforce all of these invariants, IBOS puts all network processes in their own protection domains. By putting network processes in their own protection domains, the kernel naturally ensures that network requests from web page instances and Ethernet frames from the NIC are routed to the correct network process. The third invariant is that the Ethernet frames from network processes to the NIC must have an IP address and TCP port that matches the origin of the network process. IBOS kernel checks all outgoing Ethernet frames before sending them to the NIC to check the IP address and TCP port against the label of the sending network process. The fourth invariant is HTTP data from network processes to web page instances must adhere to the SOP. The IBOS kernel inspects HTTP
data before forwarding it to the appropriate web page instance and drops any HTML documents from different origins. The last invariant is that network processes for different web page instances must remain isolated by labelling the origin of the web page instance.

As for the UI subsystem, IBOS kernel enforces 3 invariants. The first is the browser and web page content displays are isolated by using a frame buffer video driver and page protections to isolate portions of the screen. The second invariant is that only the current tab can access the screen, mouse, and keyboard by mapping the frame buffer memory region and routing mouse and keyboard events into the currently active web page instance. the final invariant is the URL of the current tab is displayed to the user by using the kernel display area to display the URL.

The IBOS kernel also offers custom security by allowing web sites to specify a server-side policy file that IBOS retrieves to restrict network accesses for a web page instance.

==Critique==

===Structure===
This paper was very well organized and executed. It naturally flows and keeps order in what it is trying to explain without the need to flip back and reference another piece of content in the paper. Starting with the core mechanics of why it is needed to how the kernel is organized and working its way up to many high-level pieces of information it felt like a natural progression of ideas, giving you the information you need to understand upcoming concepts.

===Evaluation===
The evaluation of the IBOS security has some flaws,it is not very thorough and the data set the testing against is potentially confounding.

The IBOS has shown through internal testing that it is able to resist 77% of attacks from a set of 175 security bugs whereas Chrome is only able prevent 46%. The improvement sounds impressive however, the set of security bugs they tested against was obtained from Google “Chrome’s bug tracker”. The fact they are comparing known security flaws in Chrome against the new IBOS makes their improvement of 31% far less impressive.

In addition, Their initial test set contained 217 bugs with duplicates removed, and 42 bugs were omitted because they were denial of service attacks and the IBOS does not protect against that form of attack. It is understandable this is out of the scope of this research. However, that is a big set of flaws which are not addressed.

Furthermore, the researches only compared their results against the Chrome web browser. A comparison which also includes other browsers such as Mozilla Firefox, Internet Explorer and Safari would be much more compelling.

==References==

[1] CVE - Common Vulnerabilities and Exposures (CVE). http://cve.mitre.org.

[2] Rushby, John (1981). "Design and Verification of Secure Systems". 8th ACM Symposium on Operating System Principles. Pacific Grove, California, US. pp. 12–21.

[3] B. Lampson, M. Abadi, M. Burrows and E. Wobber, Authentication in Distributed Systems: Theory and Practice, ACM Transactions on Computer Systems 1992, on page 6.

COMP 3000 Essay 2 2010 Question 2

2010-12-03T05:05:59Z

Gsmith6:

==Paper==
'''Trust and Protection in the Illinois Browser Operating System'''

http://www.usenix.org/events/osdi10/tech/full_papers/Tang.pdf

Shuo Tang, Haohui Mai, Samuel T. King

''University of Illinois at Urbana-Champaig''

Presentation slides to go along with the paper: Trust and Protection in the Illinois Browser Operating System. http://www.cs.uiuc.edu/homes/stang6/ibos.html#slide1

==Background Concepts==
In the world we are in, the web is everywhere and runs in different operating systems and browsers. The Illinois Browser Operating System (IBOS) is not just a new browser to improve security, it is also a full operating system. It was developed by three graduate students at the University of Illinois. It’s main goal is to expose browser-level abstractions at the lowest possible software layer, reducing the trusted computing base for web browsers. Many websites and web applications have become major targets for attackers and hackers. These attackers are always finding new ways of exploiting even the most secure systems. Just recently, cross-site scripting (XSS) has become the most common security vulnerability over the age old buffer overflow [[#References | [1]]].

Plenty of research has gone in to improving security among the various web browsers on the market today but all browsers still remain susceptible to attacks on the lower layers. Compromised Ethernet drivers can send sensitive HTTP packets to third parties, compromised storage modules can send persistent data to unwanted viewers and compromised window managers can overlay fake interfaces common in phishing attacks [[#References | [1]]]. Common web browsers run on top of commodity operating systems with shared system services and user-mode libraries, increasing the trusted computing base(TCB). IBOS looks to solve this issue by exposing browser-level abstractions rather than just general-purpose abstractions. Important concepts such as cookies, HTTP connections and tabs for displaying pages are all brought into the browser abstraction layer. By moving all of these subsystems, the IBOS system ultimately aims to reduce the computer's TCB.

===Monolithic vs Modular===

The internet today has become an important part of our lives, it is used everywhere and in everyday life. It all started with the user only being allowed to read and browse the content, but today the expansion has led to allow the ability to write our own content and have our own impact on the rest of the world. The introduction of Web 2.0 brought upon an enormous change to the web, as well as introducing the concept of security concerns. Modern web applications are mostly designed to meet a newer modular architecture.

Modularly designed architecture browsers such as Chrome are designed in a way that each browsing instance is assigned to a unique operating system process. It is designed to perform much better in terms of fault-tolerance, accountability, security, memory management and performance. Chrome has the ability to be able to run even when another web program crashes. Memory management in a modular architecture is designed to handle each process at a time and when the program closes the memory space is ready to be used by another program. Scheduling for modular browsers handled at the OS level and web programs are able to run parallel.

Monolithic architectures browser such as Firefox, are monolithic and easy to exploit, since they run in a single address space. It is designed in a way that all the components for the web run in a single process. Disadvantages about monolithic architectures are that if a single web program crashes, then all of the other web browsers also crash. This can cause all of the data to be lost that is currently saved. Memory management is also considered poor in a monolithic architecture because it allocates memory at the beginning of the web program and may contain leaks. This is a major problem, because when the program finishes execution, the memory leak will have grown exponentially.

===TCB===

The TCB is the hardware and software that is critical to the computer's security. It's a combination of kernel and trusted processes made of hardware, firmware, and software that are critical to a computer's security [[#References | [2]]]. In the words of Lampson et al., TCB is defined as:

''"a small amount of software and hardware that security depends on and that we distinguish from a much larger amount that can misbehave without affecting security."'' [[#References | [3]]].

Modern operating system-browser combinations have massive TCBs that may have several millions of lines of code. By extracting components such as device drivers from the kernel, one can lower a systems TCB considerably. If a device driver is outside of the TCB and becomes corrupted, the effects would not be too severe, but if the driver is left in the TCB, then the results could be cataclysmic. By removing elements from the TCB, it becomes smaller, thereby reducing the risk of having an attack get inside.

==Research Problem==
Modern browsers, such as Google Chrome and Mozilla Firefox, are continuously being revised and updated to keep ahead of the latest attacks, but continuously have hundreds of security vulnerabilities. Most of these attacks are simple, mostly harmless assaults on web applications, but many attacks are on the browser or even the operating system and its libraries. Since the browser runs lower on the shared storage stack, a successful attack on a browser can have horrible repercussions because it gives access to all of the browser data for all of the web applications. It also provides the attacker with access to other resources on the system which is being exploited. An attack on the operating system can be disastrous if it is successful and may cause serious damage to the entire system. This is partly because the attackers can access arbitrary states and events, allowing them to have full control over the system. The focus of this research is to prevent and decrease the attacks on the browser, libraries, operating systems and system services.

==Contribution==

===Architecture and Design===
The authors have developed IBOS to reduce security risks, without compromising speed and efficiency. One of the ways they have achieved this is through their implementation of process creation. Essentially there are two types of processes. A web page instance and a traditional process. Any time the user opens a new tab, clicks on a link, or enters a web address in the uniform resource locator(URL) bar, the IBOS kernel creates a new process. Upon creating a web page instance process, the kernel labels it with the originating address of the HTTP request. If a web site such as ''facebook.com'' decides to host an outside script, also known as an iframe, from another website, the kernel creates a new process for the embedded script and labels it appropriately. Traditional processes are every other process that is created for the local machine. These processes are simply labeled as ''localhost''.

An IBOS label contains a protocol, domain and port. By creating unique labels for each web page instance, the kernel can isolate them from one another. This prevents a compromised component from taking control of other processes. Also by labeling where requests come from, the IBOS kernel can ensure that the data it is receiving is in fact from the expected origin. Each time a process makes a request, the kernel checks to ensure that the process is in fact allowed to access the requested resources.

Each time an HTTP request comes in over the network, the network process renders the HTTP into a TCP stream, which is in turn converted into a series of Ethernet frames that are sent to the network interface controller(NIC). By doing all of this, the process is excluded from accessing any data structure. The process must first pass through the kernel before it receives what it has requested.

IBOS has considerably smaller TCB compared to other modern browsers. Where both Chrome and Firefox come in at over 4 million plus lines of code in their trusted computing base, IBOS has only about 42,000. Since IBOS isolates each process, it was also able to prevent between 75-100% of vulnerabilities from affected components on a machine. Using Chrome, the researchers tested 175 known issues on the IBOS kernel which ranged from memory exploits to interface spoofing. Out of all the known issues, IBOS was able to prevent 135 or 77% of the issues whereas Chrome was only able to contain 83 of them. The issue is that Chrome is able to catch exploits in its rendering engine since it is in a sandbox but any exploits that took advantage of the browser kernel could not be prevented. This is not a problem for IBOS because many of the browser components inside the trusted computing base in Chrome have been brought outside of the IBOS TCB limiting what can be done with exploitation. IBOS also contains tabs abstraction which multiplexes the display between different web pages instances. Input devices are programmed to route to a visible tab, which prevents keylogger from hijacking another instance's session.

===Performance===
In terms of performance, IBOS is comparable to the two best performing web browsers currently released: Firefox and Chrome. For websites such as Google Maps and Facebook, IBOS actually performs much better than Firefox while loading pages. This may be due partly to the fact that IBOS was developed with the WebKit engine, which has been optimized to run Google Maps. For Facebook and Wikipedia, sites that use many HTTP requests, IBOS performs slightly slower than the other two browsers, but for the others, where there are only a few HTTP requests, IBOS runs just as quickly as the others.

IBOS have the same functionality as Firefox and Chrome which uses the standard Web kit but it gives better performance.

===Security===
In addition to labels, IBOS has other approaches at reaching their goals in security. Security variants are a way for IBOS to enforce their subsystems to perform their designated functions without interfering with other subsystems. To reach this goal, they designed invariants for different components such as driver, storage, network processes, and UI invariants.

For driver invariants, IBOS insures drivers do not access DMA buffer directly and devices can only access validated DMA buffers. their approach is to separate the the management of device control registers from the use of device buffers. Using this separation, processes can fill device specific buffers for DMA transfers, and the IBOS kernel instructs the driver to enforce the device to use a verified DMA buffer.

In storage invariant IBOS encrypts all objects before passing them to the storage subsystem. That way All of the key-value pairs maintain confidentiality and integrity even if the storage stack itself becomes compromised.

The network process invariants is where the security is focused. The first two invariants are: The kernel must route network requests from web page instances to the proper network process, and the kernel must route Ethernet frames from the NIC to the proper network processes. To enforce all of these invariants, IBOS puts all network processes in their own protection domains. By putting network processes in their own protection domains, the kernel naturally ensures that network requests from web page instances and Ethernet frames from the NIC are routed to the correct network process. The third invariant is that the Ethernet frames from network processes to the NIC must have an IP address and TCP port that matches the origin of the network process. IBOS kernel checks all outgoing Ethernet frames before sending them to the NIC to check the IP address and TCP port against the label of the sending network process. The fourth invariant is HTTP data from network processes to web page instances must adhere to the SOP. The IBOS kernel inspects HTTP
data before forwarding it to the appropriate web page instance and drops any HTML documents from different origins. The last invariant is that network processes for different web page instances must remain isolated by labelling the origin of the web page instance.

As for the UI subsystem, IBOS kernel enforces 3 invariants. The first is the browser and web page content displays are isolated by using a frame buffer video driver and page protections to isolate portions of the screen. The second invariant is that only the current tab can access the screen, mouse, and keyboard by mapping the frame buffer memory region and routing mouse and keyboard events into the currently active web page instance. the final invariant is the URL of the current tab is displayed to the user by using the kernel display area to display the URL.

The IBOS kernel also offers custom security by allowing web sites to specify a server-side policy file that IBOS retrieves to restrict network accesses for a web page instance.

==Critique==

===Structure===
This paper was very well organized and executed. It naturally flows and keeps order in what it is trying to explain without the need to flip back and reference another piece of content in the paper. Starting with the core mechanics of why it is needed to how the kernel is organized and working its way up to many high-level pieces of information it felt like a natural progression of ideas, giving you the information you need to understand upcoming concepts.

===Evaluation===
The evaluation of the IBOS security has some flaws,it is not very thorough and the data set the testing against is potentially confounding.

The IBOS has shown through internal testing that it is able to resist 77% of attacks from a set of 175 security bugs whereas Chrome is only able prevent 46%. The improvement sounds impressive however, the set of security bugs they tested against was obtained from Google “Chrome’s bug tracker”. The fact they are comparing known security flaws in Chrome against the new IBOS makes their improvement of 31% far less impressive.

In addition, Their initial test set contained 217 bugs with duplicates removed, and 42 bugs were omitted because they were denial of service attacks and the IBOS does not protect against that form of attack. It is understandable this is out of the scope of this research. However, that is a big set of flaws which are not addressed.

Furthermore, the researches only compared their results against the Chrome web browser. A comparison which also includes other browsers such as Mozilla Firefox, Internet Explorer and Safari would be much more compelling.

==References==

[1] CVE - Common Vulnerabilities and Exposures (CVE). http://cve.mitre.org.

[2] Rushby, John (1981). "Design and Verification of Secure Systems". 8th ACM Symposium on Operating System Principles. Pacific Grove, California, US. pp. 12–21.

[3] B. Lampson, M. Abadi, M. Burrows and E. Wobber, Authentication in Distributed Systems: Theory and Practice, ACM Transactions on Computer Systems 1992, on page 6.

COMP 3000 Essay 2 2010 Question 2

2010-12-03T04:16:26Z

Gsmith6:

==Paper==
'''Trust and Protection in the Illinois Browser Operating System'''

http://www.usenix.org/events/osdi10/tech/full_papers/Tang.pdf

Shuo Tang, Haohui Mai, Samuel T. King

''University of Illinois at Urbana-Champaig''

Presentation slides to go along with the paper: Trust and Protection in the Illinois Browser Operating System. http://www.cs.uiuc.edu/homes/stang6/ibos.html#slide1

==Background Concepts==
In the world we are in, the web is everywhere and runs in different operating systems and browsers. The Illinois Browser Operating System (IBOS) is not just a new browser to improve security, it is also a full operating system. It was developed by three graduate students at the University of Illinois. It’s main goal is to expose browser-level abstractions at the lowest possible software layer, reducing the trusted computing base for web browsers. Many websites and web applications have become major targets for attackers and hackers. These attackers are always finding new ways of exploiting even the most secure systems. Just recently, cross-site scripting (XSS) has become the most common security vulnerability over the age old buffer overflow [[#References | [1]]].

Plenty of research has gone in to improving security among the various web browsers on the market today but all browsers still remain susceptible to attacks on the lower layers. Compromised Ethernet drivers can send sensitive HTTP packets to third parties, compromised storage modules can send persistent data to unwanted viewers and compromised window managers can overlay fake interfaces common in phishing attacks [[#References | [1]]]. Common web browsers run on top of commodity operating systems with shared system services and user-mode libraries, increasing the trusted computing base(TCB). IBOS looks to solve this issue by exposing browser-level abstractions rather than just general-purpose abstractions. Important concepts such as cookies, HTTP connections and tabs for displaying pages are all brought into the browser abstraction layer. By moving all of these subsystems, the IBOS system ultimately aims to reduce the computer's TCB.

===Monolithic vs Modular===

The internet today has become an important part of our lives, it is used everywhere and in everyday life. It all started with the user only being allowed to read and browse the content, but today the expansion has led to allow the ability to write our own content and have our own impact on the rest of the world. The introduction of Web 2.0 brought upon an enormous change to the web, as well as introducing the concept of security concerns. Modern web applications are mostly designed to meet a newer modular architecture.

Modularly designed architecture browsers such as Chrome are designed in a way that each browsing instance is assigned to a unique operating system process. It is designed to perform much better in terms of fault-tolerance, accountability, security, memory management and performance. Chrome has the ability to be able to run even when another web program crashes. Memory management in a modular architecture is designed to handle each process at a time and when the program closes the memory space is ready to be used by another program. Scheduling for modular browsers handled at the OS level and web programs are able to run parallel.

Monolithic architectures browser such as Firefox, are monolithic and easy to exploit, since they run in a single address space. It is designed in a way that all the components for the web run in a single process. Disadvantages about monolithic architectures are that if a single web program crashes, then all of the other web browsers also crash. This can cause all of the data to be lost that is currently saved. Memory management is also considered poor in a monolithic architecture because it allocates memory at the beginning of the web program and may contain leaks. This is a major problem, because when the program finishes execution, the memory leak will have grown exponentially.

===TCB===

The TCB is the hardware and software that is critical to the computer's security. It's a combination of kernel and trusted processes made of hardware, firmware, and software that are critical to a computer's security [[#References | [2]]]. In the words of Lampson et al., TCB is defined as:

''"a small amount of software and hardware that security depends on and that we distinguish from a much larger amount that can misbehave without affecting security."'' [[#References | [3]]].

Modern operating system-browser combinations have massive TCBs that may have several millions of lines of code. By extracting components such as device drivers from the kernel, one can lower a systems TCB considerably. If a device driver is outside of the TCB and becomes corrupted, the effects would not be too severe, but if the driver is left in the TCB, then the results could be cataclysmic. By removing elements from the TCB, it becomes smaller, thereby reducing the risk of having an attack get inside.

==Research Problem==
Modern browsers, such as Google Chrome and Mozilla Firefox, are continuously being revised and updated to keep ahead of the latest attacks, but continuously have hundreds of security vulnerabilities. Most of these attacks are simple, mostly harmless assaults on web applications, but many attacks are on the browser or even the operating system and its libraries. Since the browser runs lower on the shared storage stack, a successful attack on a browser can have horrible repercussions because it gives access to all of the browser data for all of the web applications. It also provides the attacker with access to other resources on the system which is being exploited. An attack on the operating system can be disastrous if it is successful and may cause serious damage to the entire system. This is partly because the attackers can access arbitrary states and events, allowing them to have full control over the system. The focus of this research is to prevent and decrease the attacks on the browser, libraries, operating systems and system services.

==Contribution==

===Architecture and Design===
The authors have developed IBOS to reduce security risks, without compromising speed and efficiency. One of the ways they have achieved this is through their implementation of process creation. Essentially there are two types of processes. A web page instance and a traditional process. Any time the user opens a new tab, clicks on a link, or enters a web address in the uniform resource locator(URL) bar, the IBOS kernel creates a new process. Upon creating a web page instance process, the kernel labels it with the originating address of the HTTP request. If a web site such as ''facebook.com'' decides to host an outside script, also known as an iframe, from another website, the kernel creates a new process for the embedded script and labels it appropriately. Traditional processes are every other process that is created for the local machine. These processes are simply labeled as ''localhost''.

An IBOS label contains a protocol, domain and port. By creating unique labels for each web page instance, the kernel can isolate them from one another. This prevents a compromised component from taking control of other processes. Also by labeling where requests come from, the IBOS kernel can ensure that the data it is receiving is in fact from the expected origin. Each time a process makes a request, the kernel checks to ensure that the process is in fact allowed to access the requested resources.

IBOS has considerably smaller TCB compared to other modern browsers. Where both Chrome and Firefox come in at over 4 million plus lines of code in their trusted computing base, IBOS has only about 42,000. Since IBOS isolates each process, it was also able to prevent between 75-100% of vulnerabilities from affected components on a machine. Using Chrome, the researchers tested 175 known issues on the IBOS kernel which ranged from memory exploits to interface spoofing. Out of all the known issues, IBOS was able to prevent 135 or 77% of the issues whereas Chrome was only able to contain 83 of them. The issue is that Chrome is able to catch exploits in its rendering engine since it is in a sandbox but any exploits that took advantage of the browser kernel could not be prevented. This is not a problem for IBOS because many of the browser components inside the trusted computing base in Chrome have been brought outside of the IBOS TCB limiting what can be done with exploitation. IBOS also contains tabs abstraction which multiplexes the display between different web pages instances. Input devices are programmed to route to a visible tab, which prevents keylogger from hijacking another instance's session.

===Performance===
In terms of performance, IBOS is comparable to the two best performing web browsers currently released: Firefox and Chrome. For websites such as Google Maps and Facebook, IBOS actually performs much better than Firefox while loading pages. This may be due partly to the fact that IBOS was developed with the WebKit engine, which has been optimized to run Google Maps. For Facebook and Wikipedia, sites that use many HTTP requests, IBOS performs slightly slower than the other two browsers, but for the others, where there are only a few HTTP requests, IBOS runs just as quickly as the others.

IBOS have the same functionality as Firefox and Chrome which uses the standard Web kit but it gives better performance.

===Security===
In addition to labels, IBOS has other approaches at reaching their goals in security. Security variants are a way for IBOS to enforce their subsystems to perform their designated functions without interfering with other subsystems. To reach this goal, they designed invariants for different components such as driver, storage, network processes, and UI invariants.

For driver invariants, IBOS insures drivers do not access DMA buffer directly and devices can only access validated DMA buffers. their approach is to separate the the management of device control registers from the use of device buffers. Using this separation, processes can fill device specific buffers for DMA transfers, and the IBOS kernel instructs the driver to enforce the device to use a verified DMA buffer.

In storage invariant IBOS encrypts all objects before passing them to the storage subsystem. That way All of the key-value pairs maintain confidentiality and integrity even if the storage stack itself becomes compromised.

The network process invariants is where the security is focused. The first two invariants are: The kernel must route network requests from web page instances to the proper network process, and the kernel must route Ethernet frames from the NIC to the proper network processes. To enforce all of these invariants, IBOS puts all network processes in their own protection domains. By putting network processes in their own protection domains, the kernel naturally ensures that network requests from web page instances and Ethernet frames from the NIC are routed to the correct network process. The third invariant is that the Ethernet frames from network processes to the NIC must have an IP address and TCP port that matches the origin of the network process. IBOS kernel checks all outgoing Ethernet frames before sending them to the NIC to check the IP address and TCP port against the label of the sending network process. The fourth invariant is HTTP data from network processes to web page instances must adhere to the SOP. The IBOS kernel inspects HTTP
data before forwarding it to the appropriate web page instance and drops any HTML documents from different origins. The last invariant is that network processes for different web page instances must remain isolated by labelling the origin of the web page instance.

As for the UI subsystem, IBOS kernel enforces 3 invariants. The first is the browser and web page content displays are isolated by using a frame buffer video driver and page protections to isolate portions of the screen. The second invariant is that only the current tab can access the screen, mouse, and keyboard by mapping the frame buffer memory region and routing mouse and keyboard events into the currently active web page instance. the final invariant is the URL of the current tab is displayed to the user by using the kernel display area to display the URL.

The IBOS kernel also offers custom security by allowing web sites to specify a server-side policy file that IBOS retrieves to restrict network accesses for a web page instance.

==Critique==

===Structure===
This paper was very well organized and executed. It naturally flows and keeps order in what it is trying to explain without the need to flip back and reference another piece of content in the paper. Starting with the core mechanics of why it is needed to how the kernel is organized and working its way up to many high-level pieces of information it felt like a natural progression of ideas, giving you the information you need to understand upcoming concepts.

===Evaluation===
The evaluation of the IBOS security has some flaws,it is not very thorough and the data set the testing against is potentially confounding.

The IBOS has shown through internal testing that it is able to resist 77% of attacks from a set of 175 security bugs whereas Chrome is only able prevent 46%. The improvement sounds impressive however, the set of security bugs they tested against was obtained from Google “Chrome’s bug tracker”. The fact they are comparing known security flaws in Chrome against the new IBOS makes their improvement of 31% far less impressive.

In addition, Their initial test set contained 217 bugs with duplicates removed, and 42 bugs were omitted because they were denial of service attacks and the IBOS does not protect against that form of attack. It is understandable this is out of the scope of this research. However, that is a big set of flaws which are not addressed.

Furthermore, the researches only compared their results against the Chrome web browser. A comparison which also includes other browsers such as Mozilla Firefox, Internet Explorer and Safari would be much more compelling.

==References==

[1] CVE - Common Vulnerabilities and Exposures (CVE). http://cve.mitre.org.

[2] Rushby, John (1981). "Design and Verification of Secure Systems". 8th ACM Symposium on Operating System Principles. Pacific Grove, California, US. pp. 12–21.

[3] B. Lampson, M. Abadi, M. Burrows and E. Wobber, Authentication in Distributed Systems: Theory and Practice, ACM Transactions on Computer Systems 1992, on page 6.

COMP 3000 Essay 2 2010 Question 2

2010-12-03T04:01:42Z

Gsmith6:

==Paper==
'''Trust and Protection in the Illinois Browser Operating System'''

http://www.usenix.org/events/osdi10/tech/full_papers/Tang.pdf

Shuo Tang, Haohui Mai, Samuel T. King

''University of Illinois at Urbana-Champaig''

Presentation slides to go along with the paper: Trust and Protection in the Illinois Browser Operating System. http://www.cs.uiuc.edu/homes/stang6/ibos.html#slide1

==Background Concepts==
In the world we are in, the web is everywhere and runs in different operating systems and browsers. The Illinois Browser Operating System (IBOS) is not just a new browser to improve security, it is also a full operating system. It was developed by three graduate students at the University of Illinois. It’s main goal is to expose browser-level abstractions at the lowest possible software layer, reducing the trusted computing base for web browsers. Many websites and web applications have become major targets for attackers and hackers. These attackers are always finding new ways of exploiting even the most secure systems. Just recently, cross-site scripting (XSS) has become the most common security vulnerability over the age old buffer overflow [[#References | [1]]].

Plenty of research has gone in to improving security among the various web browsers on the market today but all browsers still remain susceptible to attacks on the lower layers. Compromised Ethernet drivers can send sensitive HTTP packets to third parties, compromised storage modules can send persistent data to unwanted viewers and compromised window managers can overlay fake interfaces common in phishing attacks [[#References | [1]]]. Common web browsers run on top of commodity operating systems with shared system services and user-mode libraries, increasing the trusted computing base(TCB). IBOS looks to solve this issue by exposing browser-level abstractions rather than just general-purpose abstractions. Important concepts such as cookies, HTTP connections and tabs for displaying pages are all brought into the browser abstraction layer. By moving all of these subsystems, the IBOS system ultimately aims to reduce the computer's TCB.

===Monolithic vs Modular===

The internet today has become an important part of our lives, it is used everywhere and in everyday life. It all started with the user only being allowed to read and browse the content, but today the expansion has led to allow the ability to write our own content and have our own impact on the rest of the world. The introduction of Web 2.0 brought upon an enormous change to the web, as well as introducing the concept of security concerns. Modern web applications are mostly designed to meet a newer modular architecture.

Modularly designed architecture browsers such as Chrome are designed in a way that each browsing instance is assigned to a unique operating system process. It is designed to perform much better in terms of fault-tolerance, accountability, security, memory management and performance. Chrome has the ability to be able to run even when another web program crashes. Memory management in a modular architecture is designed to handle each process at a time and when the program closes the memory space is ready to be used by another program. Scheduling for modular browsers handled at the OS level and web programs are able to run parallel.

Monolithic architectures browser such as Firefox, are monolithic and easy to exploit, since they run in a single address space. It is designed in a way that all the components for the web run in a single process. Disadvantages about monolithic architectures are that if a single web program crashes, then all of the other web browsers also crash. This can cause all of the data to be lost that is currently saved. Memory management is also considered poor in a monolithic architecture because it allocates memory at the beginning of the web program and may contain leaks. This is a major problem, because when the program finishes execution, the memory leak will have grown exponentially.

===TCB===

The TCB is the hardware and software that is critical to the computer's security. It's a combination of kernel and trusted processes made of hardware, firmware, and software that are critical to a computer's security [[#References | [2]]]. In the words of Lampson et al., TCB is defined as:

''"a small amount of software and hardware that security depends on and that we distinguish from a much larger amount that can misbehave without affecting security."'' [[#References | [3]]].

Modern operating system-browser combinations have massive TCBs that may have several millions of lines of code. By extracting components such as device drivers from the kernel, one can lower a systems TCB considerably. If a device driver is outside of the TCB and becomes corrupted, the effects would not be too severe, but if the driver is left in the TCB, then the results could be cataclysmic. By removing elements from the TCB, you make it smaller, thereby reducing the risk of having an attack get inside.

==Research Problem==
Modern browsers, such as Google Chrome and Mozilla Firefox, are continuously being revised and updated to keep ahead of the latest attacks, but continuously have hundreds of security vulnerabilities. Most of these attacks are simple, slightly harmful assaults on web applications, but many attacks are on the browser or even the operating system and its libraries. Since the browser runs lower on the shared storage stack, a successful attack on a browser can have horrible repercussions because it gives access to all of the browser data for all of the web application. It also provides the attacker with access to other resources on the system which is being exploited. An attack on the operating system can be disastrous if it is successful and may cause serious damage to the entire system because the attackers can access arbitrary states and events, allowing them to have full control over the system. The focus of this research is to prevent and decrease the attacks on the browser, libraries, operating systems and system services.

==Contribution==

===Architecture and Design===
The authors have developed IBOS to reduce security risks, without compromising speed and efficiency which leads to reduced TCB. One of the ways they have achieved this is through the use of process creation. Essentially there are two types of processes. A web page instance and a traditional process. Any time the user opens a new tab, clicks on a link, or enters a web address in the uniform resource locator(URL) bar, the IBOS kernel creates a new process. Upon creating a web page instance process, the kernel labels it with the originating address of the HTTP request. If a web site such as ''facebook.com'' decides to host an outside script, also known as an iframe, from another website, the kernel creates a new process for the embedded script and labels it appropriately. Traditional processes are every other process that is created for the local machine. These processes are simply labeled as ''localhost''.

An IBOS label contains a protocol, domain and port. By creating unique labels for each web page instance, the kernel can isolate them from one another. This prevents a compromised component from taking control of other processes. Also by labeling where requests come from, the IBOS kernel can ensure that the data it is receiving is in fact from the expected origin.

IBOS has considerably smaller TCB compared to other modern browsers. Where both Chrome and Firefox come in at over 4 million plus lines of code in their trusted computing base, IBOS has only about 42,000. Since IBOS isolates each process, it was also able to prevent between 75-100% of vulnerabilities from affected components on a machine. Using Chrome, the researchers tested 175 known issues on the IBOS kernel which ranged from memory exploits to interface spoofing. Out of all the known issues, IBOS was able to prevent 135 or 77% of the issues whereas Chrome was only able to contain 83 of them. The issue is that Chrome is able to catch exploits in its rendering engine since it is in a sandbox but any exploits that took advantage of the browser kernel could not be prevented. This is not a problem for IBOS because many of the browser components inside the trusted computing base in Chrome have been brought outside of the IBOS TCB limiting what can be done with exploitation. IBOS also contains tabs abstraction which multiplexes the display between different web pages instances. Input devices are programmed to route to a visible tab, which prevents keylogger from hijacking another instance's session.

===Performance===
In terms of performance, IBOS is comparable to the two best performing web browsers currently released: Firefox and Chrome. For websites such as Google Maps and Facebook, IBOS actually performs much better than Firefox while loading pages. This may be due partly to the fact that IBOS was developed with the WebKit engine, which has been optimized to run Google Maps. For Facebook and Wikipedia, sites that use many HTTP requests, IBOS performs slightly slower than the other two browsers, but for the others, where there are only a few HTTP requests, IBOS runs just as quickly as the others.

IBOS have the same functionality as Firefox and Chrome which uses the standard Web kit but it gives better performance.

===Security===
In addition to labels, IBOS has other approaches at reaching their goals in security. Security variants are a way for IBOS to enforce their subsystems to perform their designated functions without interfering with other subsystems. To reach this goal, they designed invariants for different components such as driver, storage, network processes, and UI invariants.

For driver invariants, IBOS insures drivers do not access DMA buffer directly and devices can only access validated DMA buffers. their approach is to separate the the management of device control registers from the use of device buffers. Using this separation, processes can fill device specific buffers for DMA transfers, and the IBOS kernel instructs the driver to enforce the device to use a verified DMA buffer. In storage invariant IBOS encrypts all objects before passing them to the storage subsystem. That way All of our key-value pairs maintain confidentiality and integrity even if the storage stack itself becomes compromised.

==Critique==

===Structure===
This paper was very well organized and executed. It naturally flows and keeps order in what it is trying to explain without the need to flip back and reference another piece of content in the paper. Starting with the core mechanics of why it is needed to how the kernel is organized and working its way up to many high-level pieces of information it felt like a natural progression of ideas, giving you the information you need to understand upcoming concepts.

===Evaluation===
The evaluation of the IBOS security has some flaws,it is not very thorough and the data set the testing against is potentially confounding.

The IBOS has shown through internal testing that it is able to resist 77% of attacks from a set of 175 security bugs whereas Chrome is only able prevent 46%. The improvement sounds impressive however, the set of security bugs they tested against was obtained from Google “Chrome’s bug tracker”. The fact they are comparing known security flaws in Chrome against the new IBOS makes their improvement of 31% far less impressive.

In addition, Their initial test set contained 217 bugs with duplicates removed, and 42 bugs were omitted because they were denial of service attacks and the IBOS does not protect against that form of attack. It is understandable this is out of the scope of this research. However, that is a big set of flaws which are not addressed.

Furthermore, the researches only compared their results against the Chrome web browser. A comparison which also includes other browsers such as Mozilla Firefox, Internet Explorer and Safari would be much more compelling.

==References==

[1] CVE - Common Vulnerabilities and Exposures (CVE). http://cve.mitre.org.

[2] Rushby, John (1981). "Design and Verification of Secure Systems". 8th ACM Symposium on Operating System Principles. Pacific Grove, California, US. pp. 12–21.

[3] B. Lampson, M. Abadi, M. Burrows and E. Wobber, Authentication in Distributed Systems: Theory and Practice, ACM Transactions on Computer Systems 1992, on page 6.

COMP 3000 Essay 2 2010 Question 2

2010-12-02T20:02:58Z

Gsmith6:

COMP 3000 Essay 2 2010 Question 2

2010-12-02T19:35:40Z

Gsmith6:

COMP 3000 Essay 2 2010 Question 2

2010-12-02T16:27:40Z

Gsmith6: /* Contribution */

COMP 3000 Essay 2 2010 Question 2

2010-12-02T16:15:52Z

Gsmith6: /* Background Concepts */

COMP 3000 Essay 2 2010 Question 2

2010-12-02T16:15:37Z

Gsmith6:

Talk:COMP 3000 Essay 2 2010 Question 2

2010-12-02T16:14:37Z

Gsmith6: /* Comments & Discussion */

=Comments & Discussion=
I've added to the Contribution part of the essay. I've basically explained as much as I thought was pertinent in what the section was asking for but don't be shy to add more!

--[[User:Selliot3|Selliot3]] 14:11, 2 December 2010 (UTC)

IBOS is designed to talk directly to the hardware. That's why they have everything at the lower level. All that extra TCB in other browsers was for all the extra stuff like services, OS components blah blah. I get their drift and how less code is secure but I don't get how they did it! From where do they get the services they claim to have taken out but still operates the same as any other browser. There is a catch somewhere but I can't find it in the paper or maybe I'm blind. I've been reading a lot of text but I got nowhere, its either too complex or not close to what I'm looking for.

--[[User:Ymoussou|Youcef M.]] 04:19, 2 December 2010 (UTC)

Brief description of the research problem below. (Still Needs expanding/fleshing out. can anyone help expand on why exactly shrinking the TCB will be more secure. I'm fuzzy on that)

The IBOS attempts to improve the security of web browsers. The writers argue that the large size of the trusted code bases (TCB) which modern web browsers make use of increases the possibility of a security hole. For example a hijacked window manager could be used to draw a fake phishing website overtop a web browser. The researchers solution is drastically shrinking the size of the TCB. The TCB is shrunk by turning the web browser into an operating system in itself with direct access to hardware abstractions.

Meanwhile, the IBOS must still support existing web applications while maintaining security.

[[User:Atubman|Atubman]] 03:36, 2 December 2010 (UTC)

This is what I understand of the TCB. Basically it is all of the components that are essential to the security of the computer system. Pretty much the stuff inside the kernel that if is compromised is "bad news bears". By removing things like device drivers, you reduce the TCB quite considerably (like probably 10s of thousands of lines of code). The smaller the TCB, the less of a chance that you have of an essential component getting corrupted. I don't know if I'm explaining it right, but maybe someone else can expand on what I've written here.
--[[User:Gsmith6|Gsmith6]] 16:13, 2 December 2010 (UTC)

EDIT: I've pretty much explained the background concept behind IBOS and I kind of added the way it's executed near the end. Feel free to move that into the research section.

I can work on the background of IBOS

--[[User:Selliot3|Selliot3]] 23:03, 22 November 2010 (UTC)

It seems we only have 5/7 members. We should start splitting up the tasks and assign who gets what. So if everybody writes what section they would like to work on that would be great.

--[[User:Ymoussou|Youcef M.]] 15:19, 20 November 2010 (UTC)

I'll do the contribution section. I'll be reading through the paper thoroughly today and taking notes as I go. I'll post them later on this page as a sort of cheat-sheet/reminder. --[[User:Gsmith6|Gsmith6]] 17:45, 25 November 2010 (UTC)

=Group Members=

Leave your name and e-mail address if you are assigned to this question.

[[User:Ymoussou|Youcef M.]] moussoud@gmail.com

I am alive and still in the class, selliot3@connect.carleton.ca

--[[User:Selliot3|Selliot3]] 18:12, 15 November 2010 (UTC)

Still in the class, andrewtubman84@gmail.com

[[User:Atubman|Atubman]]

I'm here. I have received an email reply from John Vanden Heuvel as well (he may not see this) gsmith0413@gmail.com
--[[User:Gsmith6|Gsmith6]] 22:31, 15 November 2010 (UTC)

[[User:vviveka2|vG]]

I am here... and replied to the email

=Question 2 members=

Elliott Charles selliot3

Moussoud Youcef ymoussou

Pharand Alexandre apharan2

Smith Geoffrey gsmith6

Tubman Andrew atubman

Vanden Heuvel John jvheuvel

Vivekanandarajah Vijitharan vviveka2

=Raw Information=

The web itself is ubiquitous which a person can use for communication; banking, business, social networking and it can be useful for other purposes. There are different type of vulnerabilities web applications, browser, OS and library vulnerabilities. Insecure web browsers are monolithic, and they are easy to exploit. Secure web browser such as chrome isolate web applications and it still contain huge trusted computing base (TCB). Browser abstractions as the first-class OS, contains reduced TCB for web browser and it also have protection to withstand attacks to most components. [[User:vviveka2|vG]]

=Extra Resources=
http://www.cs.uiuc.edu/homes/stang6/ibos.html#slide1
I found some presentation slides by Shuo Tang, Haohui Mai and Sam King, the authors and developers of IBOS
--[[User:Gsmith6|Gsmith6]] 22:35, 25 November 2010 (UTC)

Talk:COMP 3000 Essay 2 2010 Question 2

2010-12-02T16:13:16Z

Gsmith6: /* Comments & Discussion */

=Comments & Discussion=
I've added to the Contribution part of the essay. I've basically explained as much as I thought was pertinent in what the section was asking for but don't be shy to add more!

--[[User:Selliot3|Selliot3]] 14:11, 2 December 2010 (UTC)

IBOS is designed to talk directly to the hardware. That's why they have everything at the lower level. All that extra TCB in other browsers was for all the extra stuff like services, OS components blah blah. I get their drift and how less code is secure but I don't get how they did it! From where do they get the services they claim to have taken out but still operates the same as any other browser. There is a catch somewhere but I can't find it in the paper or maybe I'm blind. I've been reading a lot of text but I got nowhere, its either too complex or not close to what I'm looking for.

--[[User:Ymoussou|Youcef M.]] 04:19, 2 December 2010 (UTC)

Brief description of the research problem below. (Still Needs expanding/fleshing out. can anyone help expand on why exactly shrinking the TCB will be more secure. I'm fuzzy on that)

The IBOS attempts to improve the security of web browsers. The writers argue that the large size of the trusted code bases (TCB) which modern web browsers make use of increases the possibility of a security hole. For example a hijacked window manager could be used to draw a fake phishing website overtop a web browser. The researchers solution is drastically shrinking the size of the TCB. The TCB is shrunk by turning the web browser into an operating system in itself with direct access to hardware abstractions.

Meanwhile, the IBOS must still support existing web applications while maintaining security.

[[User:Atubman|Atubman]] 03:36, 2 December 2010 (UTC)

This is what I understand of the TCB. Basically it is all of the components that are essential to the security of the computer system. Pretty much the stuff inside the kernel that if is compromised is "bad news bears". By removing things like device drivers, you reduce the TCB quite considerably (like probably 10s of thousands of lines of code). The smaller the TCB, the less of a chance that you have an essential component getting corrupted. I don't know if I'm explaining it right, but maybe someone else can expand on what I've written here.
--[[User:Gsmith6|Gsmith6]] 16:13, 2 December 2010 (UTC)

EDIT: I've pretty much explained the background concept behind IBOS and I kind of added the way it's executed near the end. Feel free to move that into the research section.

I can work on the background of IBOS

--[[User:Selliot3|Selliot3]] 23:03, 22 November 2010 (UTC)

It seems we only have 5/7 members. We should start splitting up the tasks and assign who gets what. So if everybody writes what section they would like to work on that would be great.

--[[User:Ymoussou|Youcef M.]] 15:19, 20 November 2010 (UTC)

I'll do the contribution section. I'll be reading through the paper thoroughly today and taking notes as I go. I'll post them later on this page as a sort of cheat-sheet/reminder. --[[User:Gsmith6|Gsmith6]] 17:45, 25 November 2010 (UTC)

=Group Members=

Leave your name and e-mail address if you are assigned to this question.

[[User:Ymoussou|Youcef M.]] moussoud@gmail.com

I am alive and still in the class, selliot3@connect.carleton.ca

--[[User:Selliot3|Selliot3]] 18:12, 15 November 2010 (UTC)

Still in the class, andrewtubman84@gmail.com

[[User:Atubman|Atubman]]

I'm here. I have received an email reply from John Vanden Heuvel as well (he may not see this) gsmith0413@gmail.com
--[[User:Gsmith6|Gsmith6]] 22:31, 15 November 2010 (UTC)

[[User:vviveka2|vG]]

I am here... and replied to the email

=Question 2 members=

Elliott Charles selliot3

Moussoud Youcef ymoussou

Pharand Alexandre apharan2

Smith Geoffrey gsmith6

Tubman Andrew atubman

Vanden Heuvel John jvheuvel

Vivekanandarajah Vijitharan vviveka2

=Raw Information=

The web itself is ubiquitous which a person can use for communication; banking, business, social networking and it can be useful for other purposes. There are different type of vulnerabilities web applications, browser, OS and library vulnerabilities. Insecure web browsers are monolithic, and they are easy to exploit. Secure web browser such as chrome isolate web applications and it still contain huge trusted computing base (TCB). Browser abstractions as the first-class OS, contains reduced TCB for web browser and it also have protection to withstand attacks to most components. [[User:vviveka2|vG]]

=Extra Resources=
http://www.cs.uiuc.edu/homes/stang6/ibos.html#slide1
I found some presentation slides by Shuo Tang, Haohui Mai and Sam King, the authors and developers of IBOS
--[[User:Gsmith6|Gsmith6]] 22:35, 25 November 2010 (UTC)

COMP 3000 Essay 2 2010 Question 2

2010-12-02T04:15:48Z

Gsmith6:

=Paper=
'''Trust and Protection in the Illinois Browser Operating System'''

http://www.usenix.org/events/osdi10/tech/full_papers/Tang.pdf

Shuo Tang, Haohui Mai, Samuel T. King

''University of Illinois at Urbana-Champaig''

Presentation slides to go along with the paper: Trust and Protection in the Illinois Browser Operating System. http://www.cs.uiuc.edu/homes/stang6/ibos.html#slide1

==Research Problem==
Modern browsers, such as Google Chrome and Mozilla Firefox, are constantly being revised and updated to keep up with the latest attacks, but continuously have hundreds of security vulnerabilities. Most of these attacks are simple, slightly harmful assaults on web applications, but many attacks are on the browser or even the operating system/libraries. A successful attack on a browser can have horrible repercussion because these occur lower on the shared storage stack than the attacks on the applications. An attack on the operating system can be disastrous if it is successful and may cause serious damage to the entire system.

=Background Concepts=
The Illinois Browser Operating System (IBOS) is not just a new browser to improve security, it is also a full operating system. It’s main goal is to expose browser-level abstractions at the lowest possible software layer, reducing the trusted computing base for web browsers. Many websites and web applications have become major targets for attackers and hackers. Just recently, cross-site scripting has become the most common security vulnerability over the age old buffer overflow.

Plenty of research has gone in to improving security among the various web browsers on the market today but all browsers still remain susceptible to attacks on the lower layers. Compromised Ethernet drivers can send sensitive HTTP packets to third parties, compromised storage modules can send persistent data to unwanted viewers and compromised window managers can overlay fake interfaces common in phishing attacks. Common web browsers run on top of commodity operating systems with shared system services and user-mode libraries, increasing the trusted code base. IBOS looks to solve this issue by exposing browser-level abstractions rather than just general-purpose abstractions. Important concepts such as cookies, HTTP connections and tabs for displaying pages are all brought into the browser abstraction layer. By using all of these methods, the IBOS system ultimately aims to reduce the computers Trusted Computing Base(TCB).

===TCB===
The TCB is the hardware and software that is critical to the computer's security. Modern operating system/browser combinations have massive TCBs that may have several millions of lines of code. By extracting components such as device drivers from the kernel, one can lower a systems TCB considerably. If a device driver is outside of the TCB and becomes corrupted, the effects would not be too severe, but if the driver is left in the TCB, then the results could be cataclysmic. By removing elements from the TCB, the risk of having an attack get inside is greatly reduced.

=Contribution=
What are the research contribution(s) of this work? Specifically, what are the key research results, and what do they mean? (What was implemented? Why is it any better than what came before?)

=Critique=
This paper was very well organized and executed. It naturally flows and keeps order in what it is trying to explain without the need to flip back and reference another piece of content in the paper. Starting with the core mechanics of why it is needed to how the kernel is organized and working its way up to many high-level pieces of information it felt like a natural progression of ideas, giving you the information you need to understand upcoming concepts.

'''!! Don't forget to erase this !!''' ''What is good and not-so-good about this paper? You may discuss both the style and content; be sure to ground your discussion with specific references. Simple assertions that something is good or bad is not enough - you must explain why.''

=References=
You will almost certainly have to refer to other resources; please cite these resources in the style of citation of the papers assigned (inlined numbered references). Place your bibliographic entries in this section.

COMP 3000 Essay 2 2010 Question 2

2010-12-02T03:45:37Z

Gsmith6: /* TCB */

=Paper=
'''Trust and Protection in the Illinois Browser Operating System'''

http://www.usenix.org/events/osdi10/tech/full_papers/Tang.pdf

Shuo Tang, Haohui Mai, Samuel T. King

''University of Illinois at Urbana-Champaig''

Presentation slides to go along with the paper: Trust and Protection in the Illinois Browser Operating System. http://www.cs.uiuc.edu/homes/stang6/ibos.html#slide1

=Background Concepts=
The Illinois Browser Operating System (IBOS) is not just a new browser to improve security, it is also a full operating system. It’s main goal is to expose browser-level abstractions at the lowest possible software layer, reducing the trusted computing base for web browsers. Many websites and web applications have become major targets for attackers and hackers. Just recently, cross-site scripting has become the most common security vulnerability over the age old buffer overflow.

Plenty of research has gone in to improving security among the various web browsers on the market today but all browsers still remain susceptible to attacks on the lower layers. Compromised Ethernet drivers can send sensitive HTTP packets to third parties, compromised storage modules can send persistent data to unwanted viewers and compromised window managers can overlay fake interfaces common in phishing attacks. Common web browsers run on top of commodity operating systems with shared system services and user-mode libraries, increasing the trusted code base. IBOS looks to solve this issue by exposing browser-level abstractions rather than just general-purpose abstractions. Important concepts such as cookies, HTTP connections and tabs for displaying pages are all brought into the browser abstraction layer. By using all of these methods, the IBOS system ultimately aims to reduce the computers Trusted Computing Base(TCB).

===TCB===
The TCB is the hardware and software that is critical to the computer's security. Modern operating system/browser combinations have massive TCBs that may have several millions of lines of code. By extracting components such as device drivers from the kernel, one can lower a systems TCB considerably. If a device driver is outside of the TCB and becomes corrupted, the effects would not be too severe, but if the driver is left in the TCB, then the results could be cataclysmic. By removing elements from the TCB, the risk of having an attack get inside is greatly reduced.

=Research problem=
What is the research problem being addressed by the paper? How does this problem relate to past related work?

=Contribution=
What are the research contribution(s) of this work? Specifically, what are the key research results, and what do they mean? (What was implemented? Why is it any better than what came before?)

=Critique=
This paper was very well organized and executed. It naturally flows and keeps order in what it is trying to explain without the need to flip back and reference another piece of content in the paper. Starting with the core mechanics of why it is needed to how the kernel is organized and working its way up to many high-level pieces of information it felt like a natural progression of ideas, giving you the information you need to understand upcoming concepts.

'''!! Don't forget to erase this !!''' ''What is good and not-so-good about this paper? You may discuss both the style and content; be sure to ground your discussion with specific references. Simple assertions that something is good or bad is not enough - you must explain why.''

=References=
You will almost certainly have to refer to other resources; please cite these resources in the style of citation of the papers assigned (inlined numbered references). Place your bibliographic entries in this section.

COMP 3000 Essay 2 2010 Question 2

2010-12-02T03:45:08Z

Gsmith6: /* Background Concepts */

=Paper=
'''Trust and Protection in the Illinois Browser Operating System'''

http://www.usenix.org/events/osdi10/tech/full_papers/Tang.pdf

Shuo Tang, Haohui Mai, Samuel T. King

''University of Illinois at Urbana-Champaig''

Presentation slides to go along with the paper: Trust and Protection in the Illinois Browser Operating System. http://www.cs.uiuc.edu/homes/stang6/ibos.html#slide1

=Background Concepts=
The Illinois Browser Operating System (IBOS) is not just a new browser to improve security, it is also a full operating system. It’s main goal is to expose browser-level abstractions at the lowest possible software layer, reducing the trusted computing base for web browsers. Many websites and web applications have become major targets for attackers and hackers. Just recently, cross-site scripting has become the most common security vulnerability over the age old buffer overflow.

Plenty of research has gone in to improving security among the various web browsers on the market today but all browsers still remain susceptible to attacks on the lower layers. Compromised Ethernet drivers can send sensitive HTTP packets to third parties, compromised storage modules can send persistent data to unwanted viewers and compromised window managers can overlay fake interfaces common in phishing attacks. Common web browsers run on top of commodity operating systems with shared system services and user-mode libraries, increasing the trusted code base. IBOS looks to solve this issue by exposing browser-level abstractions rather than just general-purpose abstractions. Important concepts such as cookies, HTTP connections and tabs for displaying pages are all brought into the browser abstraction layer. By using all of these methods, the IBOS system ultimately aims to reduce the computers Trusted Computing Base(TCB).

==TCB==
The TCB is the hardware and software that is critical to the computer's security. Modern operating system/browser combinations have massive TCBs that may have several millions of lines of code. By extracting components such as device drivers from the kernel, one can lower a systems TCB considerably. If a device driver is outside of the TCB and becomes corrupted, the effects would not be too severe, but if the driver is left in the TCB, then the results could be cataclysmic. By removing elements from the TCB, the risk of having an attack get inside is greatly reduced.

=Research problem=
What is the research problem being addressed by the paper? How does this problem relate to past related work?

=Contribution=
What are the research contribution(s) of this work? Specifically, what are the key research results, and what do they mean? (What was implemented? Why is it any better than what came before?)

=Critique=
This paper was very well organized and executed. It naturally flows and keeps order in what it is trying to explain without the need to flip back and reference another piece of content in the paper. Starting with the core mechanics of why it is needed to how the kernel is organized and working its way up to many high-level pieces of information it felt like a natural progression of ideas, giving you the information you need to understand upcoming concepts.

'''!! Don't forget to erase this !!''' ''What is good and not-so-good about this paper? You may discuss both the style and content; be sure to ground your discussion with specific references. Simple assertions that something is good or bad is not enough - you must explain why.''

=References=
You will almost certainly have to refer to other resources; please cite these resources in the style of citation of the papers assigned (inlined numbered references). Place your bibliographic entries in this section.

COMP 3000 Essay 2 2010 Question 2

2010-12-02T03:22:51Z

Gsmith6: /* Paper */

=Paper=
'''Trust and Protection in the Illinois Browser Operating System'''

http://www.usenix.org/events/osdi10/tech/full_papers/Tang.pdf

Shuo Tang, Haohui Mai, Samuel T. King

''University of Illinois at Urbana-Champaig''

Presentation slides to go along with the paper: Trust and Protection in the Illinois Browser Operating System. http://www.cs.uiuc.edu/homes/stang6/ibos.html#slide1

=Background Concepts=
The Illinois Browser Operating System (IBOS) is not just a new browser to improve security, it is also a full operating system. It’s main goal is to expose browser-level abstractions at the lowest possible software layer, reducing the trusted computing base for web browsers. Many websites and web applications have become major targets for attackers and hackers. Just recently, cross-site scripting has become the most common security vulnerability over the age old buffer overflow.

Plenty of research has gone in to improving security among the various web browsers on the market today but all browsers still remain susceptible to attacks on the lower layers. Compromised Ethernet drivers can send sensitive HTTP packets to third parties, compromised storage modules can send persistent data to unwanted viewers and compromised window managers can overlay fake interfaces common in phishing attacks. Common web browsers run on top of commodity operating systems with shared system services and user-mode libraries, increasing the trusted code base. IBOS looks to solve this issue by exposing browser-level abstractions rather than just general-purpose abstractions. Important concepts such as cookies, HTTP connections and tabs for displaying pages are all brought into the browser abstraction layer.

=Research problem=
What is the research problem being addressed by the paper? How does this problem relate to past related work?

=Contribution=
What are the research contribution(s) of this work? Specifically, what are the key research results, and what do they mean? (What was implemented? Why is it any better than what came before?)

=Critique=
This paper was very well organized and executed. It naturally flows and keeps order in what it is trying to explain without the need to flip back and reference another piece of content in the paper. Starting with the core mechanics of why it is needed to how the kernel is organized and working its way up to many high-level pieces of information it felt like a natural progression of ideas, giving you the information you need to understand upcoming concepts.

'''!! Don't forget to erase this !!''' ''What is good and not-so-good about this paper? You may discuss both the style and content; be sure to ground your discussion with specific references. Simple assertions that something is good or bad is not enough - you must explain why.''

=References=
You will almost certainly have to refer to other resources; please cite these resources in the style of citation of the papers assigned (inlined numbered references). Place your bibliographic entries in this section.

COMP 3000 Essay 2 2010 Question 2

2010-12-02T03:22:16Z

Gsmith6: /* References */

=Paper=
'''Trust and Protection in the Illinois Browser Operating System'''

http://www.usenix.org/events/osdi10/tech/full_papers/Tang.pdf

Shuo Tang, Haohui Mai, Samuel T. King

''University of Illinois at Urbana-Champaig''

=Background Concepts=
The Illinois Browser Operating System (IBOS) is not just a new browser to improve security, it is also a full operating system. It’s main goal is to expose browser-level abstractions at the lowest possible software layer, reducing the trusted computing base for web browsers. Many websites and web applications have become major targets for attackers and hackers. Just recently, cross-site scripting has become the most common security vulnerability over the age old buffer overflow.

Plenty of research has gone in to improving security among the various web browsers on the market today but all browsers still remain susceptible to attacks on the lower layers. Compromised Ethernet drivers can send sensitive HTTP packets to third parties, compromised storage modules can send persistent data to unwanted viewers and compromised window managers can overlay fake interfaces common in phishing attacks. Common web browsers run on top of commodity operating systems with shared system services and user-mode libraries, increasing the trusted code base. IBOS looks to solve this issue by exposing browser-level abstractions rather than just general-purpose abstractions. Important concepts such as cookies, HTTP connections and tabs for displaying pages are all brought into the browser abstraction layer.

=Research problem=
What is the research problem being addressed by the paper? How does this problem relate to past related work?

=Contribution=
What are the research contribution(s) of this work? Specifically, what are the key research results, and what do they mean? (What was implemented? Why is it any better than what came before?)

=Critique=
This paper was very well organized and executed. It naturally flows and keeps order in what it is trying to explain without the need to flip back and reference another piece of content in the paper. Starting with the core mechanics of why it is needed to how the kernel is organized and working its way up to many high-level pieces of information it felt like a natural progression of ideas, giving you the information you need to understand upcoming concepts.

'''!! Don't forget to erase this !!''' ''What is good and not-so-good about this paper? You may discuss both the style and content; be sure to ground your discussion with specific references. Simple assertions that something is good or bad is not enough - you must explain why.''

=References=
You will almost certainly have to refer to other resources; please cite these resources in the style of citation of the papers assigned (inlined numbered references). Place your bibliographic entries in this section.

COMP 3000 Essay 2 2010 Question 2

2010-12-02T03:18:10Z

Gsmith6: /* References */

=Paper=
'''Trust and Protection in the Illinois Browser Operating System'''

http://www.usenix.org/events/osdi10/tech/full_papers/Tang.pdf

Shuo Tang, Haohui Mai, Samuel T. King

''University of Illinois at Urbana-Champaig''

=Background Concepts=
The Illinois Browser Operating System (IBOS) is not just a new browser to improve security, it is also a full operating system. It’s main goal is to expose browser-level abstractions at the lowest possible software layer, reducing the trusted computing base for web browsers. Many websites and web applications have become major targets for attackers and hackers. Just recently, cross-site scripting has become the most common security vulnerability over the age old buffer overflow.

Plenty of research has gone in to improving security among the various web browsers on the market today but all browsers still remain susceptible to attacks on the lower layers. Compromised Ethernet drivers can send sensitive HTTP packets to third parties, compromised storage modules can send persistent data to unwanted viewers and compromised window managers can overlay fake interfaces common in phishing attacks. Common web browsers run on top of commodity operating systems with shared system services and user-mode libraries, increasing the trusted code base. IBOS looks to solve this issue by exposing browser-level abstractions rather than just general-purpose abstractions. Important concepts such as cookies, HTTP connections and tabs for displaying pages are all brought into the browser abstraction layer.

=Research problem=
What is the research problem being addressed by the paper? How does this problem relate to past related work?

=Contribution=
What are the research contribution(s) of this work? Specifically, what are the key research results, and what do they mean? (What was implemented? Why is it any better than what came before?)

=Critique=
This paper was very well organized and executed. It naturally flows and keeps order in what it is trying to explain without the need to flip back and reference another piece of content in the paper. Starting with the core mechanics of why it is needed to how the kernel is organized and working its way up to many high-level pieces of information it felt like a natural progression of ideas, giving you the information you need to understand upcoming concepts.

'''!! Don't forget to erase this !!''' ''What is good and not-so-good about this paper? You may discuss both the style and content; be sure to ground your discussion with specific references. Simple assertions that something is good or bad is not enough - you must explain why.''

=References=
You will almost certainly have to refer to other resources; please cite these resources in the style of citation of the papers assigned (inlined numbered references). Place your bibliographic entries in this section.

Trust and Protection in the Illinois Browser Operating System. http://www.cs.uiuc.edu/homes/stang6/ibos.html#slide1

Talk:COMP 3000 Essay 2 2010 Question 2

2010-12-02T01:59:56Z

Gsmith6:

Talk:COMP 3000 Essay 2 2010 Question 2

2010-12-02T01:58:02Z

Gsmith6: /* Raw Information */

=Comments & Discussion=

EDIT: I've pretty much explained the background concept behind IBOS and I kind of added the way it's executed near the end. Feel free to move that into the research section.

I can work on the background of IBOS

--[[User:Selliot3|Selliot3]] 23:03, 22 November 2010 (UTC)

It seems we only have 5/7 members. We should start splitting up the tasks and assign who gets what. So if everybody writes what section they would like to work on that would be great.

--[[User:Ymoussou|Youcef M.]] 15:19, 20 November 2010 (UTC)

I'll do the contribution section. I'll be reading through the paper thoroughly today and taking notes as I go. I'll post them later on this page as a sort of cheat-sheet/reminder. --[[User:Gsmith6|Gsmith6]] 17:45, 25 November 2010 (UTC)

=Group Members=

Leave your name and e-mail address if you are assigned to this question.

[[User:Ymoussou|Youcef M.]] moussoud@gmail.com

I am alive and still in the class, selliot3@connect.carleton.ca

--[[User:Selliot3|Selliot3]] 18:12, 15 November 2010 (UTC)

Still in the class, andrewtubman84@gmail.com

[[User:Atubman|Atubman]]

I'm here. I have received an email reply from John Vanden Heuvel as well (he may not see this) gsmith0413@gmail.com
--[[User:Gsmith6|Gsmith6]] 22:31, 15 November 2010 (UTC)

[[User:vviveka2|vG]]

I am here... and replied to the email

=Question 2 members=

Elliott Charles selliot3

Moussoud Youcef ymoussou

Pharand Alexandre apharan2

Smith Geoffrey gsmith6

Tubman Andrew atubman

Vanden Heuvel John jvheuvel

Vivekanandarajah Vijitharan vviveka2

=Raw Information=

The web itself is ubiquitous which a person can use for communication; banking, business, social networking and it can be useful for other purposes. There are different type of vulnerabilities web applications, browser, OS and library vulnerabilities. Insecure web browsers are monolithic, and they are easy to exploit. Secure web browser such as chrome isolate web applications and it still contain huge trusted computing base (TCB). Browser abstractions as the first-class OS, contains reduced TCB for web browser and it also have protection to withstand attacks to most components. [[User:vviveka2|vG]]

==Rough Notes==

Essay Notes
-most attacks are on web apps, but some attack the browser itself
-if the attack on the browser is successful, it can be more devestating because the attacker can access much more of the system (lower on the computer stack than web apps)
-even more serious if an attacker gets lower in the stack (i.e. libraries, OSes)
-higher level vulnerabilities are more common, but low levels mean more devastating attacks
-new browsers create a seperation between functionality of browser, and security mechanisms, but they still share certain services and libraries that attackers can use to infiltrate the system.
-Illinois Browser Operating System (IBOS) is an effort to reduce the TCB of web browsers.
-achieves this by taking out shared OS components and services from the browser's TCB.
-These include device drivers, network protocol implementations, storage stack, and window-management software.
-they are put into the browser level abstractions.
-traditional methods are to put application-specific abstractions on top of the OS abstractions.

CONTRIBUTIONS:
-first system to improve broswer and OS security by making browser-level abstractions into first class OS abstractions, creating seperation between browser functionality and browser security
-by having low-layer software expose browser abstractions allows for removal of almost all traditional OS components from the TCB
-IBOS still allows traditional applications to interact with the browser and shared OS services without compromising security.

ARCHITECTURE:
-uses microkernel, exokernel, and safety kernel concepts
-by making the security decisions at the lowest layer of software, the authors avoid putting the millions of lines of library and OS code in the TCB.
-monitors the sharing of data between web apps and traditional apps
-built to enforce browser policies of existing systems
-IBOS kernel acts similarily to traditional microkernels, it manages global resources and creates new processes.
-All messages being passed travel through the reference monitor, where it is subjected to the overall security policy.
-HTTP requests are reformated into a TCP stream which then gets converted into a series of Ethernet frames.
-by storing HTTP cookies and HTML5 local storage objects using the storage manager, they can be examined by the reference monitor.
-by using namespaces, the OS isolates objects such as web apps, traditional apps and device drivers
-two different types of processes, web page instances and traditional processes
-new process, called a web page instance, is created for each individual web page that the user visits
-these are resposnsible for issuing HTTP reguests, parsing HTML, executing JavaScript, and rendering web content to a tab
-the key difference between the two types of processes is the security label the IBOS kernel gives them
-plugins are run as traditional processes, but they are launched by the browser and are given access to browser states

CURRENT BROWSER POLICIES
-same-origin policy (SOP) isolates web pages that come from different origins
-modern browsers have their SOP scattered throught millions of lines of code, and are often troublesome to implement
-iframes can be labeled by the page developers as from the same origin as the hosting page, or from different origins

IBOS SECURITY POLICIES AND MECHANISMS
-want to be able to ensure the IBOS kernel upholds security policies even if any number of subsystems have been compromised
-susceptible components include all drivers, browser API managers, web page instances, and traditional processes.
-IBOS labels specify the resources that a process can access or messages it can receive
-web page instances are labeled by the origin of the HTML document
-traditional processes are labeled as "localhost"
-IBOS labels the processes upon creation, and keeps the labels unchanged throughout the processes life
-because the IBOS kernel labels processes itself, rather than the process labelling itself, it ensures it has the correct label information.
-by using these security invariants, they can extracting security relevant information from messages automatically, they can remove almost all of the components found in modern operating systems from the TCB, including device drivers
-SI 0: All components can only perform their designated functions
-SI 1: Drivers cannot access DMA buffers directly
-SI 2: Devices can only access validated DMA buffers
-seperates management of device control registers from the use of the device buffers to allow checking that the communications are within the IBOS security policies
-SI 3: All of the key-value pairs maintain confidentiality and integrity even if the storage stack becomes compromised
-this is acheived by encrypting objects before passing them to the storage subsystem
-SI 4: The kernel must route network requests from web page instances to the proper network process
-SI 5: The kernel must route Ethernet frames from the NIC to the proper netwok processes
-SI 6: Ethernet frames from network processes to the NIC must have an IP address and TCP port that matches the origin of the network process
-SI 7: HTTP data from network process to web page instances must adhere to the SOP
-SI 8: Network processes for different web page instances must remain isolated
-all network processes are put into their own protection domains
-using the labeling system, the kernel can ensure that network requests from web page instances and Ethernet frames from the NIC are routed to the correct network process
-by cross checking the outgoing requests with the sending network process, the IBOS kernel ensures that the NIC send the request to the right host, and that it is sending the right data
-SI 9: The browser chrome and web page content displays are isolated
-SI 10: Only the current tab can access the screen, mouse and keyboard
-SI 11: The URL of the current tab is displayed to the user
-sections of the IBOS display are isolated from one another
-3 horizontal sections, each with its own access privelages
-top bar is only for the IBOS kernel, middle section is reserved for the UI subsystem to draw the browser chrome, and the rest is for the web page instance
-by displaying the current URL in the top section, the kernel cross-references that URL with the URL the web page instance claims to be
-this is a simple task, but is not currently implemented right in modern browsers
-by running iframes in a seperate web page instance, the kernel can examine the label, and determine if it is safe
-IBOS cannot stop XSS (cross-site scripting) but it does try and prevent the attack from causing any serious damage

IMPLEMENTATION
-divided into 3 parts. The IBOS kernel, the IBOS messaging passing interfaces, and the IBOS subsystems
-IBOS kernel is implemented on top of the L4Ka::Pistachio microkernel
-scheduler is a static priority system
-entire IBOS TCB has 42,044 lines of code compared to:
-Firefox on Linux > 5,684,639
-ChromeOS > 4,407,066
-looked at 28 known Linux vulnerabilities, and were able to prevent 27 of them for a success rate of 96%
-also looked at Chrome's 295 bugs, 42 of which cause denial-of-service, which IBOS does not address currently
-78 were not actual security issues
-175 were legit issues, and can be grouped into 7 categories
-Memory Exploitation = IBOS contained or eliminated slightly more problems than Chrome contained
-XSS = IBOS was able to eliminate or contain all issues known in Chrome
-SOP circumvention = also stopped all issues, vs. Chrome did not contain any
-Sandbox bypassing = all vs. none
-Interface spoofing = all vs. none
-UI design flaw = neither browser was able to contain or eliminate any of these issues
-Miscellaneous = IBOS only managed to stop 14% of these problems
-Overall, Chrome was able to contain 46%, while IBOS either contained or eliminated 77% of these known issues
-Chrome can contain most issues that get into the rendering engine, but cannot contain those that are in the browser kernel
-a compromised Ethernet driver cannot access the DMA buffers
-a compromised storage module has little impact on the data confidentiality and integrity because the IBOS kernel encrypts all data, only thing it can do is delete objects
-a compromised network stack is constrained as well because the IBOS kernel ensures that each network process can only send relevant data to the expected host
-a compromised window manager cannot affect other subsystems, because IBOS does not allow it to do anything but draw the browser chrome.
-compared IBOS, IBOS-Linux, Firefox, and Chrome loading several different websites
-overall, Chrome was the fastest
-IBOS, and IBOS-Linux were as fast as Chrome when loading web pages with few HTTP requests, but web pages like facebook, and wikipedia (which use a lot of HTTP requests, slowed IBOS and IBOS-Linux considerably)

ADDITIONAL RELATED WORK
-ChromeOS and IBOS have fundamentally differnt design philosophies
-ChromeOS is implemented on top of the pre-existing Linux kernel, but will make modifications to the kernel to strip out uneeded portions
-IBOS starts with a clean slate and only includes the system functionality for the system.
-by starting from scratch, IBOS is able to keep the lines of code down in the TCB by 2 to 3 orders of magnitude compared to ChromeOS
--[[User:Gsmith6|Gsmith6]] 01:58, 2 December 2010 (UTC)

=Extra Resources=
http://www.cs.uiuc.edu/homes/stang6/ibos.html#slide1
I found some presentation slides by Shuo Tang, Haohui Mai and Sam King, the authors and developers of IBOS
--[[User:Gsmith6|Gsmith6]] 22:35, 25 November 2010 (UTC)

Talk:COMP 3000 Essay 2 2010 Question 2

2010-11-25T22:41:14Z

Gsmith6: /* Extra Resources */

Talk:COMP 3000 Essay 2 2010 Question 2

2010-11-25T22:35:23Z

Gsmith6: /* Question 2 members */

Talk:COMP 3000 Essay 2 2010 Question 2

2010-11-25T17:45:13Z

Gsmith6: /* Comments & Discussion */

Talk:COMP 3000 Essay 2 2010 Question 2

2010-11-15T22:31:59Z

Gsmith6: /* Group Members */

=Group Members=

Leave your name and e-mail address if you are assigned to this question.

[[User:Ymoussou|Youcef M.]] moussoud@gmail.com

I am alive and still in the class, selliot3@connect.carleton.ca

--[[User:Selliot3|Selliot3]] 18:12, 15 November 2010 (UTC)

Still in the class, andrewtubman84@gmail.com

[[User:Atubman|Atubman]]

I'm here. I have received an email reply from John Vanden Heuvel as well (he may not see this) gsmith0413@gmail.com
--[[User:Gsmith6|Gsmith6]] 22:31, 15 November 2010 (UTC)

=Question 2 members=

Elliott Charles selliot3

Moussoud Youcef ymoussou

Pharand Alexandre apharan2

Smith Geoffrey gsmith6

Tubman Andrew atubman

Vanden Heuvel John jvheuvel

Vivekanandarajah Vijitharan vviveka2

Talk:COMP 3000 Essay 1 2010 Question 10

2010-10-15T07:38:32Z

Gsmith6:

Hey all,

I think we should write down our emails here so we can further discuss stuff without having to login here.
('''***Note that discussions over email can't be counted towards your participation grade!***'''--[[User:Soma|Anil]])

Geoff Smith (gsmith0413@gmail.com) - gsmith6

Andrew Bujáki (abujaki [at] Connect or Live.ca)
***I'm usually on MSN(Live) for collaboration at nights, Just make sure to put in a little message about who you are when you're adding me. :)

I used Google Scholar and came to this page http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=812717&tag=1#
Which briefly touches on the issues of Flash memory. Specifically, inability to update in place, and limited write/erase cycles.

Inability to update in place could refer to the way the flash disk is programmed, instead of bit-by-bit, it is programmed block-by-block. A block would have to be erased and completely reprogrammed in order to flip one bit after it's been set.
http://en.wikipedia.org/wiki/Flash_memory#Block_erasure

Limited write/erase: Flash memory typically has a short lifespan if it's being used a lot. Writing and erasing the memory (Changing, updating, etc) Will wear it out. Flash memory has a finite amount of writes, (varying on manufacturer, models, etc), and once they've been used up, you'll get bad sectors, corrupt data, and generally be SOL.
http://en.wikipedia.org/wiki/Flash_memory#Memory_wear

Filesystems would have to be changed to play nicely with these constraints, where it must use blocks efficiently and nicely, and minimize writing/erasing as much as possible.

I found a paper that talks about the performance, capabilities and limitations of NAND flash storage.

Abstract: "This presentation provides an in-depth examination of the
fundamental theoretical performance, capabilities, and
limitations of NAND Flash-based Solid State Storage (SSS). The
tutorial will explore the raw performance capabilities of NAND
Flash, and limitations to performance imposed by mitigation of
reliability issues, interfaces, protocols, and technology types.
Best practices for system integration of SSS will be discussed.
Performance achievements will be reviewed for various
products and applications. "

Link: http://www.flashmemorysummit.com/English/Collaterals/Proceedings/2009/20090812_T1B_Smith.pdf

There's no Starting place like Wikipedia, even if you shouldn't source it.

http://en.wikipedia.org/wiki/Flash_Memory

http://en.wikipedia.org/wiki/LogFS

http://en.wikipedia.org/wiki/Hard_disk

http://en.wikipedia.org/wiki/Wear_leveling

http://en.wikipedia.org/wiki/Hot_spot_%28computer_science%29

http://en.wikipedia.org/wiki/Solid-state_drive

Hey Guys,

We really don't have much time to get this done. Lets meet tomorrow after class and get our bearings to do this properly.

Fedor

A few of us have Networking immediately after class. I know personally I won't be able to make anything set on Tuesday.
Additionally, he spoke briefly about hotspots on the disk for our question last week, where places on the disk would be written to far more often than others.
As well, for bibliographical citing, http://bibme.org is a wonderful resource for the popular formats (I.e. MLA). If it should come down to that.
~Andrew

===links===

Start Posting some stuff to source from:

http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1199079&tag=1
--"Introduction to flash memory"

http://portal.acm.org/citation.cfm?id=1244248
--"Wear Leveling" (it's about a proposed way of doing it, but explains a whole bunch of other things to do that)

http://portal.acm.org/citation.cfm?id=1731355
--"Online maintenance of very large random samples on flash storage" (ie dealing with the constraints of Flash Storage in a system that might actually be written to 100000 times)

http://vlsi.kaist.ac.kr/paper_list/2006_TC_CFFS.pdf
--"An Efficient NAND Flash File System for Flash Memory Storage" discuses shortcomings of using hard disk based file systems and current flash based file systems

http://maltiel-consulting.com/NAND_vs_NOR_Flash_Memory_Technology_Overview_Read_Write_Erase_speed_for_SLC_MLC_semiconductor_consulting_expert.pdf
--"NAND vs NOR Flash Memory" (note: i didn't get this off of Google scholar but it seems to be written by someone from Toshiba. is that ok?)

Hi everybody,

So here are the latest news. Geoff, Andrew and myself had a meeting after class today and came up with a plan for writing this thing.

We decided to have 3 parts:

1. What flash storage is, why its good but also why it must have the problems that it does (the assumption is that it must have them, why would it otherwise?)
[don't know much about this just now... basics include that there is NOR (reads slightly faster)and NAND (holds more, writes faster, erases much faster, lasts about ten times longer) flash with NAND being especially popular for storage (what's NOR good for?). Here, we'd ideally want to talk about why flash was invented (supposed as an alternative to slow ROM), why it was suitable for that, and how it works on a technical level. Then, we'd want to mention why this technical functionality was innovative and useful but also why it came with two serious set-backs: having a limited-number of re-write cycles and needing to erase a block at a time.]

Either way, Flash storage affords far faster fetch times than the traditional platter-based HDD, and stability of information in a sense. Where the data is not actually stored, but reprogrammed, in a sense, the data is more secure and is less likely to be erased easily. On that note, in order to flip a single bit, that entire block will need to be erased, then reprogrammed. In an 'old' HDD, let's say, When the HDD fails at the end of its life cycle, your data is gone. (unless you're willing to shell out $200/hr to have it recovered, yes I've seen companies in Ottawa that do this.) In a flash HDD, when it reaches the end of its life, it merely becomes read-only. Bugger for Databases, but useful for technical notes and archives, let's say.
With today's modern gaming computers, Flash memory can be good on quick load times, however with limited read-writes, it could afford better use to things that are not updated as frequently. I.e... Well I don't have a better example than a webserver hosting a company's CSS and scripts.
~Source: Years in the 'biz

Flash memory started out as a replacement for EPROMs. At the time EPROMs needed a UV photoemission to be erased while flash memory could be erased electronically. The first flash memory product came out in 1988 but it did not take off until the late 1990’s because it could not be reliable produced. NOR and NAND memory is named after the arrangement of the cells in the memory array. NOR based flash memory benefits from having very fast burst read times but slower write times. Due to the structure of NOR memory programs stored in NOR based memory can be executed without being loaded into RAM first. NAND flash memory has a very large storage capacity and can read and write large files relatively fast. NAND is more suited for storage while NOR memory is better suited for direct program execution such as in CMOS chips.
source: http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1199079&tag=1 , http://maltiel-consulting.com/NAND_vs_NOR_Flash_Memory_Technology_Overview_Read_Write_Erase_speed_for_SLC_MLC_semiconductor_consulting_expert.pdf

2. How a traditional disk-based file-system works and why the limitations of flash storage make the two a poor match
[the obvious answer seems to be that traditional file-systems could just write to whatever memory was available but if they did this with a flash file-systems, certain chunks of memory would become unusable before others and the memory would be more difficult to work with. Also, disk-based file systems need to deal with seeking times which means that they want to organize their data in such a way as to reduce those (by putting related things together?) - with Flash, this isn't really a problem and thus one constraint the less to be concerned with.]

3. How a log based file-system works and why this method of operation is so well suited to working with flash memory especially in light of the latter's inherent limitations
[...]

At this time, the plan is that Geoff will work on #3 today, Andrew will work on #1 tomorrow and I will work on #2 tomorrow. The three of us will make an effort to consult some somewhat more painfully technical literature in order to gain insight into our respective queries. Whatever insight we find will be posted here.

Then, we will meet again on Thursday after class to decide how to actually write the essay.

PS, if there is anybody in the group besides the three of us - let us know so you can find a way to contribute to this... as at least two of us are competent essayists, painfully technical research would on one or more of the above topics would be a great way to contribute... especially if you could post it here prior to one of us going over the same thing.

Fedor

-- I'm not that great (but absolutely horrid) at essays and I'm alright at research, but if nothing else I have Thursday off and nothing (else) that needs doing by Friday so I can probably spend a bunch of time working on it just before it's due. -- ''Nick L''

-- Hay sorry I was unable to attend the meeting after class today. I am not too good at writing essays as well but I am pretty good at summarizing and researching. I am not too sure at what you would like me to do. Right now I'll assume you need me to research/summarizing articles for the 3 topics above. If you need me to do anything else post it here. I'll be checking the discussion regularly until this due. once again sorry for missing the meeting-- Paul Cox.

-- Hey i'm also supposed to be in on this. Sorry i couldn't contribute sooner because i was playing catchup in my other classes. Let me know what i can do and i'll be on it asap. - kirill (k.kashigin@gmail.com)
update: i'm gonna be helping Fedor with #2

PS, this article http://docs.google.com/viewer?a=v&q=cache:E7-H_pv_18wJ:citeseerx.ist.psu.edu/viewdoc/download%3Fdoi%3D10.1.1.92.2279%26rep%3Drep1%26type%3Dpdf+flash+memory+and+disk-based+file+systems&hl=en&gl=ca&pid=bl&srcid=ADGEESgspy-jqIdLOpaLYlPPoM56kjLPwXcL3_eMbTTBRkI7PG0jQKl9vIieTAYHubPu0EdQ0V4ccaf_p0S_SnqKMirSIM0Qoq5E0NpLd0M7LAGaE51wkD0F55cRSkX8dnTqx_9Yx2E7&sig=AHIEtbS-yfGI9Y48DJ0WyEEhmsXInelRGw looks really useful for part 3.

---same article as above but shorter link: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.160.5142

PPS, and this article looks really great for understanding how log based file systems work: http://delivery.acm.org/10.1145/150000/146943/p26-rosenblum.pdf?key1=146943&key2=3656986821&coll=GUIDE&dl=GUIDE&CFID=108397378&CFTOKEN=72657973

Hey Luc (TA) here, Anandtech ran a series of articles on solid state drives that you guys might find useful. It mostly looked at hardware aspects but it gives some interesting insights on how to modify file systems to better support flash memory.

http://www.anandtech.com/cpuchipsets/intel/showdoc.aspx?i=3403

http://www.anandtech.com/storage/showdoc.aspx?i=3531&p=1

http://anandtech.com/storage/showdoc.aspx?i=3631

--[[User:3maisons|3maisons]] 19:44, 12 October 2010 (UTC)

Hey Paul&Kirill,

If one of you guys could help me out with #2, that would be really great. I was going to work on that tomorrow, but I also have another large assignment to deal with and not having to do this research would greatly ease my life. Moreover, I do intend to work on writing&polishing the essay on Thursday as I have a lot of experience with that and it far more than research. Let me know if either one of you can help me with this.

The other person could probably read over what Luc posted for us and see if it fits into our framework. Just be sure to state who is going to do what.

Nick,

Honestly, we really hope to have the research done by Thursday. If that is the only day that you are free and you're not a writer, I'm honestly not sure what you could do. Perhaps someone else can think of something.

- Fedor

I'm gonna have something for #2 up tonight. -kirill

So I found this article on Reddit, posted from Linux Weekly News on pretty much exactly what we are looking at. It's entitled "Solid-state storage devices and the block layer"

http://lwn.net/SubscriberLink/408428/68fa8465da45967a/ --[[User:Gsmith6|Gsmith6]] 20:36, 13 October 2010 (UTC)

I wasn't exactly sure how much information i was supposed to present but here's what i got for #2:

Most conventional file systems are designed to me implemented on hard disk drives. This fact does not mean they cannot be implemented on a solid state drive (file storage that uses flash memory instead of magnetic discs). It would however, in many ways, defeat the purpose of using flash memory. The most consuming process for an HDD is seeking data by relocating the read-head and spinning the magnetic disk. A traditional file system optimizes the way it stores data by placing related blocks close-by on the disk to minimize mechanical movement within the HDD. One of the great advantages of flash memory, which accounts for its fast read speed, is that there is no need to seek data physically so there is no need to waste resources laying out the data in close proximity.
A traditional HDD file system will also attempt to defragment itself, moving blocks of data around for closer proximity on the magnetic disk. This process, although beneficial for HDD's, is harmful and inefficient for flash based storage. A flash optimal file system needs to reduce the amount of erase operations, since flash memory only has a limited amount of erase cycles as well as having very slow erase speeds.
When an HDD rewrites data to a physical location there is no need for it to erase the previously occupying data first, so a traditional disk based file system doesn't worry about erasing data from unused memory blocks. In contrast flash memory needs to first erase the data block before it can modify any of it contents. Since the erase procedure is extremely slow, its not practical to overwrite old data every time. It is also decremental to the life span of flash memory.
To maximize the potential of flash based memory the file system would have to write new data to empty memory blocks. This method would also call for some sort of garbage collection to erase unused blocks when the system is idle, which does not get implemented in conventional file systems since it is not needed.

--kirill

So Fedor and I were talking in the labs, and we came to the conclusion that we have been focusing on just the translation from a regular file system to a flash drive. We were under the impression that this was in fact the "Flash Optimized System", but pulling up some more articles, I'm finding that this is not necessarily the case.

This paper here http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.128.6156&rep=rep1&type=pdf.
shows an example from Axis Communications where they developed a file system specifically designed to be used on flash drives.

Now I haven't completely read it, so it might just be an optimized translational system, but its at least a start.

At our meeting today, we've decided that it would be best if people could post a rough summary of their notes in the appropriate sections, and I will rewrite them into an essay, which Fedor will go through later tonight to edit and add some more information.

Paul: I missed your comment that you weren't that great at writing, and good at research. If you want some articles behind the pay-walls, I've saved a bunch of them and emailed them to myself. Just email me (at the address at the top of the page) and I'll be more than happy to send some your way.

PS. some more references

Design tradeoffs for SSD performance http://portal.acm.org/citation.cfm?id=1404014.1404019
A log buffer-based flash translation layer using fully-associative sector translation http://delivery.acm.org/10.1145/1280000/1275990/a18-lee.pdf?key1=1275990&key2=0709607821&coll=GUIDE&dl=GUIDE&CFID=105787273&CFTOKEN=74601780

--[[User:Gsmith6|Gsmith6]] 15:03, 14 October 2010 (UTC)

I don't have any notes on this computer. >: I will be adding more to my section later on tonight. Sorry. ~Andrew

Hello dudes,

Just a quick note, try to include citations in your paragraphs - each time that you make a claim which came from evidence, put a little number [X, pp. page-number (if applicable)] into your text. Then, put the same [X] at the bottom of the page with the bibliographical information about the source. The prof hasn't yet gotten back to me about his preferred citation format, so just stick with this one for now:

Authors. ''Title''. Web-page. Date of article. Web (the word). Date you accessed it.

Here's an example:

[1] Kawaguchi, Nishioka, Tamoda. ''A Flash Memory Based File System''.
http://docs.google.com/viewer?a=v&q=cache:E7-H_pv_18wJ:citeseerx.ist.psu.edu/viewdoc/download%3Fdoi%3D10.1.1.92.2279%26rep%3Drep1%26type%3Dpdf+flash+memory+and+disk-based+file+systems&hl=en&gl=ca&pid=bl&srcid=ADGEESgspy-jqIdLOpaLYlPPoM56kjLPwXcL3_eMbTTBRkI7PG0jQKl9vIieTAYHubPu0EdQ0V4ccaf_p0S_SnqKMirSIM0Qoq5E0NpLd0M7LAGaE51wkD0F55cRSkX8dnTqx_9Yx2E7&sig=AHIEtbS-yfGI9Y48DJ0WyEEhmsXInelRGw . 1995. Web. Oct. 14, 2010.

Fedor

PS, its a good idea to check this fairly frequently between now and tomorrow morning - you never know when something will come up.

Phew... for a while there I was starting to think that I had nothing about the actual "Log-Based System", but it turns out that the "Transitional Layer" is the same thing. It looks like some articles are calling it the Log system, while others are calling it the transitional layer. Pretty sure I'm going to have an experts knowledge about flash drives after reading all these articles :P
--[[User:Gsmith6|Gsmith6]] 18:13, 14 October 2010 (UTC)

Hay Geoff this is what i got so far after reading a couple of the pdfs. the double tabed points are just my annotation on how they relate to the question.

Ware leveling: p1126-chang.pdf

* Uneven wearing of flash memory due to storing data close together
* Garbage collection prefers that no blocks have pages that have data that is constantly becoming invalid
* data that remains the same for longs periods of time should be moved from block that have not be written to much and moved to blocks that haven been erased frequently.

Log file structure: 926-rosenblum.pdf

* LFS based on assumption that frequently read files will be stored in cash and that the hard disk traffic will be dominated by writes
* Writes all new info to disk in a sequential structure called a log
* Data is stored permanently in these logs no other data is stored on the hard drive
* Converts many small random synchronous writes to a large asynchronous sequential write
** Good for flash because it cuts down on writing (prolongs drive life)
** It also good because it writes to a bigger section then a page. This means it can fill a block at a time so it doesn’t fill up other blocks with random writes that would later need to be cleaned. Cuts down on cleaning.
* Inode is stored in the log on the disk while an inode map is maintained in memory which points to the inode in the hard disk. as
** This is good for flash drives because reading does not hurt the drives life and it is fast.
** This means the map will not have to be updated on the disk as frequently cutting down on the writes.
* Log systems weakness is that it is susceptible to becoming fragmented due to the larger writes.
** Since flash drives do not require fragmentation this is fine. Also since flash drives have very fast random access the system does not become bogged down when the logs are fragmented
* Log system implements a cleaning system that scans a segment in and sees if there is live data in it. If there is a certain percentage of invalid data which goes according to the cleaning policy it will be cleaned. All the live data will be copied out and the segment will be erased.
** This is a garbage collector but its built into the file system.
* Segments contain a number of blocks. Segments can contain logs or parts of logs.

I'm still sifting though the other pdfs you sent me. Would you like me to post more info or should i format this into a paragraph or 2?

--Paul

I'm sorting through my notes and putting them into a digital format. Difficult because It's hard to remember what I copied directly out of the notes for my own reference, so I'm trying to de-plagarize. I'll upload what I have to the section by 20:30, but expect more updates. ~Andrew

Hey Paul,

Thanks for helping out with the research
I'm thinking that we probably have enough information (at least for the log-based system), so maybe if you could "try" and putting them into paragraphs. I'm kind of in the middle of typing up the essay, and I'm uploading new information every 10-20 minutes or so. Even if you could get something into paragraph form, I can go through and edit, rearrange, and add stuff as I get there. --[[User:Gsmith6|Gsmith6]] 23:51, 14 October 2010 (UTC)

I'm just going to dump my entire wiki so far into my profile page, because everything's being updated so quickly.
[[User:Abujaki]] ~Andrew

I'm sitting at my computer working on much less immediately important homework if someone wants me to do something. I'm not a great writer but if there's something that needs doing I can do it. -- Nick [[User:Nlessard|Nlessard]] 00:55, 15 October 2010 (UTC)

Hey guys,

Keep up the good work!

Geoff, it would be good if you uploaded the stuff you were typing as you got each section finished. I rewrote part 2 and the intro some hours ago and I guess I could go out and write a conclusion while stuff is still being updated.

The other thing is that the current version of part 1 doesn't really seem to serve the argument - I mean, it doesn't actually tell us why flash has the constraints that it does on a technical level... and that's not surprising because that information seems to be hard to find. Still, I'm going to see if I can find out anything about that, though, so far, I haven't had much luck.

Apart from this, I'll try to get up early tomorrow to look over what's there: make sure the argument works, etc.

Good luck with the writing,

Fedor

I think it has to do with how some transistors are made. Give me a second to research that... ~Andrew

From source 20, page 269 sec 4.6.4
Stress Induced Oxide leakage
...These devices are either erased ... orboth programmed and erased by a high field (Fowler-Nordheim) injection of electrons into a very thin dielectric film to charge and discharge the floating gate. Unfortunately, [passing] this current through the oxide degrades the quality of the oxide and eventually leads to breakdown. The wearout of tunnel oxide films during high-field stress has been correlated with the buildup of both positive or negative trapped charges."... there are still some things missing from this explanation... Give me a couple more seconds... and my brain won't process this for some reason. =_= What can you guys make of this? ~Andrew

And I said NOR Flash memory had fast fetch times. I found what I was looking for, just not too sure how to explain it in text. Silicon Oxide is used for electron injection methods, including the Fowler-Nordheim Tunneling method, where it seems to say that an electric current is passed across the oxide and it thins enough for a current to be passed through a barrier into the oxide conduction... band?

Kay. New resource [19], the massively complicated theories and science would put anyone to sleep, but the basic point is The Silicon oxide layer will degrade in quality to the point where charge trapping begins to occur, the more it's used, and the more electrons that tunnel through it over time. It's like a kneaded eraser scenario. The more you use it, the poorer quality it gets until it gets all flakey and cannot do its job anymore. The same happens with the SiO2 layer in the floating gate transistors ~Andrew

hmmm... alright, I think I understand that, could you maybe try your best and add it into the essay? If not I can try and whip something up to put in there. thanks --[[User:Gsmith6|Gsmith6]] 02:39, 15 October 2010 (UTC)

Wow, I'm sorry I made you go into that... that does explain the limited erase-cycles, though. Was it the use of these materials that made flash innovative? How does that explain the block-sized erasures?

Going to go write a conclusion now.

Fedor

While looking for what you requested, I found an entire section on the oxide breakdown... oops. :P
So really nothing changes, As electrons tunnel through the oxide layer, it deteriorates. When the damage becomes too great, the oxide abruptly loses its insulative properties. (damn firefox, that's a word) Looking for the erasures now ~Andrew

Hey guys,

I'm off to sleep, but I'll be up at 5am to see what else I can do.

Fedor

Aw, I just missed you.
It turns out that erasing entire blocks is necessary based on how the chip is built. For both NOR and NAND, pairs of word lines share an erase line, which is located near the floating gate. It uses FN tunneling to erase two lines of words at a time, making the sector the smallest size you can erase simply because that's how it's physically wired.
i.e []-mem byte |-erase line

#[]|[][]|[]
#[]|[][]|[]
#[]|[][]|[]
#[]|[][]|[] ... If that makes any sense... The erase line is separate from the byte select or word select lines

Numbers for formatting sake.

I'm prolly gonna take Fedor's advice and head to bed, but I'll likely pop back on before I go to sleep in case anyone needs me to look up anything else. :P

--[[User:Lmundt|Lmundt]] 03:23, 15 October 2010 (UTC) Interesting article compaing speeds
https://noc.sara.nl/nrg/publications/RoN2010-D1.1.pdf

Wow, I can't believe I completely forgot. Hybrid Drives... I can't remember for the life of me whether they exist yet or not, but there will eventually exist a hard drive that's both flash and traditional platter. It allocates files that are not rewritten often, like an OS (well except windows and all its lovely updates) to the flash memory on install, and the remainder of the files (Documents, programs etc) to the traditional platter. Because you can't reformat an OS 100,000 times (no matter HOW badly you screw it up with viruses and the like), the flash memory will retain usability, and all the hotspots are on the platters, so you don't need to worry about minimizing disk usage.
I'm out for the night. Cheers all! ~Andrew

Alright guys... I've done as much as I can for tonight. I'm exhausted and I'm having troubles concentrating anymore. I added a few questions for people to answer as they read. I'm going to bed, hopefully people will be up early enough to fix some things that I may have missed because I sure as hell won't be :P. Really wish I could keep going though. --[[User:Gsmith6|Gsmith6]] 07:38, 15 October 2010 (UTC)

COMP 3000 Essay 1 2010 Question 10

2010-10-15T07:34:20Z

Gsmith6: /* Questions */

=Question=

How do the constraints of flash storage affect the design of flash-optimized file systems? Explain by contrasting with hard disk-based file systems.

=Answer=
First introduced in the late 80s, Flash-memory is a light, energy-independent, compact, shock-resistant and efficiently readable type of storage. It started out as a replacement for EPROMs. At the time EPROMs needed a UV photoemission to be erased while flash memory could be erased electronically.[7] Because of the particular limitations of this kind of memory, flash file systems require a fundamentally different system architecture than disk-based file-systems: these systems need to be designed in light of flash-memory’s limited number of erase-cycles and its need to conduct erasures one entire block at a time. These constraints are a direct result of the same design that gives flash its advantages with regard to [ TO WHAT?] as both are due to [TO WHAT?] . Thus, a typical disk-based file-system is not suitable for working with flash memory as it erases far too frequently and indiscriminately while being simultaneously optimized for other constraints that do not affect flash memory. This means that a different solution is necessary and that solution is the log-based file-system which is far better suited to working with flash memory because it optimizes erasures by [WHAT?].

==Flash Memory==
Flash memory is non-volatile(meaning digital storage that does not require power to retain its memory) storage space that has become more popular recently due to its fast fetch times. There are two basic forms of the flash storage system, NOR and NAND. Each type has its advantages and disadvantages. NOR has the fastest read times, but is much slower at writing. NAND on the other hand has much more capacity, faster write times, is less expensive, and has a much longer life expectancy.[2]

More and more people use flash memory, with many sizes of drives, ranging from a few hundred megabyte USB key, to a few terabyte internal solid-state drive(SSD). Two main reasons for this movement are because of flash's extremely fast read times, and its falling price. A typical flash drive has read speeds of up to 14 times faster than a hard disk drive (HDD).[17]

This extreme read speed makes flash drives a preferred method of storing games. This effectively makes loading times virtually non-existent. There is however a downside to this method. Games constantly save, modify, and change files wearing out the blocks much quicker. Flash drives have been shown be effective for the use in web-servers for running their CSS scripts or HTML pages.

Although flash drives are exponentially faster than HDDs, they still have not become the main source of data management. The reason for this is because HDDs are simply much cheaper, and flash drives still have many faults. The most critical fault is that each block in flash memory can only be erased approximately 100,000 times.[14] This poses a problem because when modifying a file, even if its a single bit, the entire block must be erased, and rewritten. This erase/rewrite slows down the write operation considerably, making it actually slower to write a file to flash than an HDD.[8]

The transistors that store the data are created with a thin strip of Silicon Oxide separating them. When the erase operation is called on the block where the transistors are located, the system fires electrons down the strip, wiping whatever bits the transistors are holding.

HDDs use a block system, in which the kernel specifies which blocks to read and write. When using a flash drive, the blocks are emulated and mapped to a physical memory address. It does through what is called a "Translational Layer".

==Traditionally Optimized File Systems==

Since the kernel asks for a block number, a conventional hard disk drive (HDD) file-system is not optimized to work with flash memory. The reason for this is that conventional hard-disks have different constraints from those of flash memory - their primary problem is to reduce seeking time, while the primary problem when working with flash memory is to erase in a minimal and balanced way.

The most consuming process for an HDD is seeking data by relocating the read-head and spinning the magnetic disk. A traditional file system optimizes the way it stores data by placing related blocks close-by on the disk in order to minimize mechanical movement within the HDD. One of the great advantages of flash memory, which accounts for its fast read speed, is that there is no need to seek data physically. This is also why defragmentaion, a procedure used by HDDs to put files into more convenient configurations and thus minimize seeking times, loses its purpose in a flash memory context. Indeed, the unnecessary erasures that it entails are both inefficient and harmful for a flash memory unit.

This comes directly out of flash memory's aforementioned constraints: the slow block-sized erasures and the limited number of erase-cycles. Because of these, a flash optimal file system needs to minimize its erase operations and also to spread out its erasures in such a way as to avoid the formation of hot-spots: sections of memory which have undergone a disproportionately high number of erasures and are thus in danger of burning out. This process of spreading out data is referred to as ''wear leveling''. To minimize hotspots, a system using flash memory would have to write new data to empty memory blocks. This method would also call for some sort of garbage collection to conduct necessary erasure operations while the system is idle. It makes better sense to do these at this time because of the slow nature of erasures in the flash memory context. Of course, there is no such feature in a traditional HDD file-system.

==Flash Optimized File Systems==

The process of wear leveling ensures that he drive does not keep erasing and writing to the same block over and over. This is achieved by writing data that doesn’t change often to blocks that have been erased frequently. Wear leveling tries to make all the blocks use up there write cycles at an even pace increasing the overall life of the hard drive.[3] This is achieved through a Log-based File System, often referred to as the Flash Transitional Layer(FTL). Essentially, the drive stores a log that keeps track of how many times each erase sector has been invalidated (or erased). The translational layer has a translation table, where each physical memory address is associated with an emulated block sectors. This allows a traditional file system that uses block sectors to be used on the flash drive. Each block has a flag which keeps track of its state. When a block is being written to, the FTL marks the blocks needed as ''allocated''. This prevents other data being written to the block that has already been allocated. The FTL then goes on to write the data in the allocated blocks. Once it completes the transaction, the system updates the allocated blocks to ''pre-valid''. Once that is completed, the drive marks the invalidated blocks to ''invalid'', while marking the newly written block as ''valid''. This entire flagging process is to ensure that the newly allocated blocks are never mixed up with the invalidated blocks.

===Banks===
The FTL organizes data using structures called banks. When the FTL gets a request to write something to memory, it uses a bank list to determine which area of the drive should be used. Essentially a bank is a group of sequential addresses, that keeps track of when it was last updated using timestamps. The FTL will only write to that bank, and once there is not enough space to write anymore, it switches out the current bank for the one with the most available space. When cleaning up the bank, the system puts it into what is called the Cleaning Bank List and removes it from the Bank List, thus avoiding any chance of some data being written to that bank while something is being erased. [8]

===Cleaner===
When the FTL realizes that there is not enough room to write new data onto the drive, it runs a garbage collection routine. This routine selects a segment to be cleaned, copies all of the valid data into a new segment, then erases everything in the old segment. This frees up the otherwise useless invalidated blocks and by not erasing every block as soon as it becomes invalidated, it saves on the amount of times that the expensive erase operation is called. The kernel can preemptively clean the drive when the system is idle.

===Why a Log File System is efficient for flash had drive===
The file system only writes a bank at a time. This means that the OS can save up the small random writes and write them all at the same time in a bank. This will cut down on the use of the expensive write command improving the overall performance of the hard drive.[9]

If a collision occurs when writing a new bank, the file system will sends the new data to an empty bank rather then erasing the existing bank and replacing it. This will cut down on using the erase function improving the life of the hard drive. Also since it only performs writes command and not a erase command and write command like a traditional file system this improves performance of the drive as well.[16]

===Systems Developed For Flash===
In 1999 a Swedish company by the name of Axis Communications, developed and released a file system that was designed specifically to be run on a flash drive. Instead of mapping each physical address space with an emulated block sector, the system creates nodes that store data. The system keeps a log of the nodes and when each node was last updated. The system also keeps track of inodes. These inodes keep a list of nodes that correspond to the relevant data. The nodes also contain information about which inode it belongs to. When the drive is mounted, the system scans all the nodes on the drive, and rebuilds the directory.
As the directory gets built, the nodes containing the data also map the physical location for each piece of data.[14]

When writing data to the drive, the nodes carrying that data get attached to the end of the log.

==Conclusion==

In this way, thanks to its [WHATEVER MAKES LOG FSs ACTUALLY GOOD AT DEALING WITH FLASH], the log-based file-system is far better suited to working with flash memory than a traditional HDD file system. The latter is utterly unfit for this task due to its placing primacy on the minimization of seeks rather than on the minimization and management of erasures. Dealing smartly with erasures is extremely important for a flash memory file system, as that memory type's particular weaknesses, the limited number of erasure cycles, the necessity to erase by the block and the relative slowness of the erasures themselves, all have to do with erasing. A good flash memory file system must therefore be built with the aim of making the best of these weaknesses and this is precisely the reason why older disk-based file systems are not suitable for flash memory while log-based file systems are. [INSPIRATIONAL LAST WORDS]

=Questions=

# Even though flash drives are exponentially faster than traditional HDDs, why are HDDs still the main method of data storage?
# Writing and erasing data are costly operations for a flash based storage drive. Why does modifying data (even a single bit) take the most amount of time?
# Why is the Flash Translational Layer so important to a flash drive's functionality? Why can you not use the traditional interface to deal with the block layer?

=References=

[1] Kim, Han-joon; Lee, Sang-goo. ''A New Flash Memory Management for Flash Storage System''. ''IEEExplore''. Dept. of Comput. Sci., Seoul Nat. Univ., 06 Aug 2002. <http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=812717&tag=1#>

[2] Smith, Lance. ''NAND Flash Solid State Storage Performance and Capability''. ''Flash Memory Summit''. SNIA Education Committee, 18 Aug 2009. <http://www.flashmemorysummit.com/English/Collaterals/Proceedings/2009/20090812_T1B_Smith.pdf>

[3] Chang, LiPin. ''On Efficient Wear Leveling for Large-Scale Flash-Memory Storage Systems''. ''Association for Computing Machinery (ACM)''. Dept. of Comput. Sci.,Nat. ChiaoTung Univ., 15 Mar 2007. <http://portal.acm.org/citation.cfm?id=1244248>

[4] Nath, Suman; Gibbons, Phillip. ''Online maintenance of very large random samples on flash storage''. ''Association for Computing Machinery (ACM)''. The VLDB Journal, 27 Jul 2007. <http://portal.acm.org/citation.cfm?id=1731355>

[5] Lim, Seung-Ho; Park; Kyu-Ho. ''An Efficient NAND Flash File System for Flash Memory Storage''. ''CORE Laboratory''. IEEE Transactions On Computers, Jul 2006. <http://vlsi.kaist.ac.kr/paper_list/2006_TC_CFFS.pdf>

[6] ''NAND vs. NOR Flash Memory Technology Overview''. ''RMG and Associates''. Toshiba America, accessed 14 Oct 2010. <http://maltiel-consulting.com/NAND_vs_NOR_Flash_Memory_Technology_Overview_Read_Write_Erase_speed_for_SLC_MLC_semiconductor_consulting_expert.pdf>

[7] Bez, Roberto; Camerlenghi, Emilio; Modelli, Alberto; Visconti, Angelo. ''Introduction to Flash Memory''. ''IEEExplore''. STMicroelectronics, 21 May 2003. <http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1199079&tag=1>

[8] Kawaguchi, Atsuo; Nishioka, Shingo; Motoda Hiroshi. ''A Flash-Memory Based File System''. ''CiteSeerX'' Advanced Research laboratory, Hitachi, Ltd., 1995. <http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.160.5142>

[9] Rosenblum, Mendel; Ouserhout, John. ''The Design and Implementation of a Log-structured File System''. ''Association for Computing Machinery (ACM)''. University of California at Berkeley, Feb 1992. <http://portal.acm.org/citation.cfm?id=146943&coll=GUIDE&dl=GUIDE&CFID=108397378&CFTOKEN=72657973&ret=1#Fulltext>

[10] Shimpi, Anand. ''Intel X25-M SSD: Intel Delivers One of the World's Fastest Drives''. ''AnAndTech''. AnAndTech, 8 Sep 2008. <http://www.anandtech.com/show/2614>

[11] Shimpi, Anand. ''The SSD Relapse: Understanding and Choosing the Best SSD''. ''AnAndTech''. AnAndTech, 30 Aug 2009. <http://www.anandtech.com/show/2829>

[12] Shimpi, Anand. ''The SSD Anthology: Understanding SSDs and New Drives from OCZ''. ''AnAndTech''. AnAndTech, 18 Mar 2009. <http://www.anandtech.com/show/2738>

[13] Corbet, Jonathan. ''Solid-State Storage Devices and the Block Layer''. ''Linux Weekly News''. Linux Weekly News, 4 Oct 2010. <http://lwn.net/Articles/408428/>

[14] Woodhouse, David. ''JFFS : The Journalling Flash File System''. ''CiteSeerX''. Red Hat, Inc, Accessed 14 Oct 2010. <http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.128.6156&rep=rep1&type=pdf>

[15] Agrawal, Nitin; Prabhakaran, Vijayan; Wobber, Ted; Davis, John; Manasse, Mark. Panigrahy, Rina. ''Design Tradeoffs for SSD Performance''. ''Association for Computing Machinery (ACM)'', USENIX 2008 Annual Technical Conference, 2008. <http://portal.acm.org/citation.cfm?id=1404014.1404019>

[16] Lee, Sang-Won, et al. ''A Log Buffer-Based Flash Translation Layer Using Fully-Associative Sector Translation''. ''Association for Computing Machinery (ACM)''. ACM Transactions on Embedded Computing Systems (TECS), Jul 2007. <http://portal.acm.org/citation.cfm?id=1275990>

[17] ''Reach New Heights in Computing Performance''. ''Micron Technology Inc''. Micro Technology Inc, Accessed 14 Oct 2010. <http://www.micron.com/products/solid_state_storage/client_ssd.html>

[18] ''Flash Memories.'' 1 ed. New York: Springer, 1999. Print.

[19] ''Nonvolatile Memory Technologies with Emphasis on Flash: A Comprehensive Guide to Understanding and Using Flash Memory Devices''. ''IEEE Press Series on Microelectronic Systems''. New York: Wiley-Ieee Press, 2008. Print.

[20] ''Nonvolatile Semiconductor Memory Technology: A Comprehensive Guide to Understanding and Using NVSM Devices''. ''IEEE Press Series on Microelectronic Systems''. New York: Wiley-Ieee Press, 1997. Print.

=External links=

Relevant Wikipedia articles: [http://en.wikipedia.org/wiki/Flash_Memory Flash Memory], [http://en.wikipedia.org/wiki/LogFS LogFS], [http://en.wikipedia.org/wiki/Hard_disk Hard Disk Drives], [http://en.wikipedia.org/wiki/Wear_leveling Wear Leveling], [http://en.wikipedia.org/wiki/Hot_spot_%28computer_science%29 Hot Spots], [http://en.wikipedia.org/wiki/Solid-state_drive Sold State Drive].

COMP 3000 Essay 1 2010 Question 10

2010-10-15T07:22:58Z

Gsmith6: /* Banks */

=Question=

How do the constraints of flash storage affect the design of flash-optimized file systems? Explain by contrasting with hard disk-based file systems.

=Answer=
First introduced in the late 80s, Flash-memory is a light, energy-independent, compact, shock-resistant and efficiently readable type of storage. It started out as a replacement for EPROMs. At the time EPROMs needed a UV photoemission to be erased while flash memory could be erased electronically.[7] Because of the particular limitations of this kind of memory, flash file systems require a fundamentally different system architecture than disk-based file-systems: these systems need to be designed in light of flash-memory’s limited number of erase-cycles and its need to conduct erasures one entire block at a time. These constraints are a direct result of the same design that gives flash its advantages with regard to [ TO WHAT?] as both are due to [TO WHAT?] . Thus, a typical disk-based file-system is not suitable for working with flash memory as it erases far too frequently and indiscriminately while being simultaneously optimized for other constraints that do not affect flash memory. This means that a different solution is necessary and that solution is the log-based file-system which is far better suited to working with flash memory because it optimizes erasures by [WHAT?].

==Flash Memory==
Flash memory is non-volatile(meaning digital storage that does not require power to retain its memory) storage space that has become more popular recently due to its fast fetch times. There are two basic forms of the flash storage system, NOR and NAND. Each type has its advantages and disadvantages. NOR has the fastest read times, but is much slower at writing. NAND on the other hand has much more capacity, faster write times, is less expensive, and has a much longer life expectancy.[2]

More and more people use flash memory, with many sizes of drives, ranging from a few hundred megabyte USB key, to a few terabyte internal solid-state drive(SSD). Two main reasons for this movement are because of flash's extremely fast read times, and its falling price. A typical flash drive has read speeds of up to 14 times faster than a hard disk drive (HDD).[17]

This extreme read speed makes flash drives a preferred method of storing games. This effectively makes loading times virtually non-existent. There is however a downside to this method. Games constantly save, modify, and change files wearing out the blocks much quicker. Flash drives have been shown be effective for the use in web-servers for running their CSS scripts or HTML pages.

Although flash drives are exponentially faster than HDDs, they still have not become the main source of data management. The reason for this is because HDDs are simply much cheaper, and flash drives still have many faults. The most critical fault is that each block in flash memory can only be erased approximately 100,000 times.[14] This poses a problem because when modifying a file, even if its a single bit, the entire block must be erased, and rewritten. This erase/rewrite slows down the write operation considerably, making it actually slower to write a file to flash than an HDD.[8]

The transistors that store the data are created with a thin strip of Silicon Oxide separating them. When the erase operation is called on the block where the transistors are located, the system fires electrons down the strip, wiping whatever bits the transistors are holding.

HDDs use a block system, in which the kernel specifies which blocks to read and write. When using a flash drive, the blocks are emulated and mapped to a physical memory address. It does through what is called a "Translational Layer".

==Traditionally Optimized File Systems==

Since the kernel asks for a block number, a conventional hard disk drive (HDD) file-system is not optimized to work with flash memory. The reason for this is that conventional hard-disks have different constraints from those of flash memory - their primary problem is to reduce seeking time, while the primary problem when working with flash memory is to erase in a minimal and balanced way.

The most consuming process for an HDD is seeking data by relocating the read-head and spinning the magnetic disk. A traditional file system optimizes the way it stores data by placing related blocks close-by on the disk in order to minimize mechanical movement within the HDD. One of the great advantages of flash memory, which accounts for its fast read speed, is that there is no need to seek data physically. This is also why defragmentaion, a procedure used by HDDs to put files into more convenient configurations and thus minimize seeking times, loses its purpose in a flash memory context. Indeed, the unnecessary erasures that it entails are both inefficient and harmful for a flash memory unit.

This comes directly out of flash memory's aforementioned constraints: the slow block-sized erasures and the limited number of erase-cycles. Because of these, a flash optimal file system needs to minimize its erase operations and also to spread out its erasures in such a way as to avoid the formation of hot-spots: sections of memory which have undergone a disproportionately high number of erasures and are thus in danger of burning out. This process of spreading out data is referred to as ''wear leveling''. To minimize hotspots, a system using flash memory would have to write new data to empty memory blocks. This method would also call for some sort of garbage collection to conduct necessary erasure operations while the system is idle. It makes better sense to do these at this time because of the slow nature of erasures in the flash memory context. Of course, there is no such feature in a traditional HDD file-system.

==Flash Optimized File Systems==

The process of wear leveling ensures that he drive does not keep erasing and writing to the same block over and over. This is achieved by writing data that doesn’t change often to blocks that have been erased frequently. Wear leveling tries to make all the blocks use up there write cycles at an even pace increasing the overall life of the hard drive.[3] This is achieved through a Log-based File System, often referred to as the Flash Transitional Layer(FTL). Essentially, the drive stores a log that keeps track of how many times each erase sector has been invalidated (or erased). The translational layer has a translation table, where each physical memory address is associated with an emulated block sectors. This allows a traditional file system that uses block sectors to be used on the flash drive. Each block has a flag which keeps track of its state. When a block is being written to, the FTL marks the blocks needed as ''allocated''. This prevents other data being written to the block that has already been allocated. The FTL then goes on to write the data in the allocated blocks. Once it completes the transaction, the system updates the allocated blocks to ''pre-valid''. Once that is completed, the drive marks the invalidated blocks to ''invalid'', while marking the newly written block as ''valid''. This entire flagging process is to ensure that the newly allocated blocks are never mixed up with the invalidated blocks.

===Banks===
The FTL organizes data using structures called banks. When the FTL gets a request to write something to memory, it uses a bank list to determine which area of the drive should be used. Essentially a bank is a group of sequential addresses, that keeps track of when it was last updated using timestamps. The FTL will only write to that bank, and once there is not enough space to write anymore, it switches out the current bank for the one with the most available space. When cleaning up the bank, the system puts it into what is called the Cleaning Bank List and removes it from the Bank List, thus avoiding any chance of some data being written to that bank while something is being erased. [8]

===Cleaner===
When the FTL realizes that there is not enough room to write new data onto the drive, it runs a garbage collection routine. This routine selects a segment to be cleaned, copies all of the valid data into a new segment, then erases everything in the old segment. This frees up the otherwise useless invalidated blocks and by not erasing every block as soon as it becomes invalidated, it saves on the amount of times that the expensive erase operation is called. The kernel can preemptively clean the drive when the system is idle.

===Why a Log File System is efficient for flash had drive===
The file system only writes a bank at a time. This means that the OS can save up the small random writes and write them all at the same time in a bank. This will cut down on the use of the expensive write command improving the overall performance of the hard drive.[9]

If a collision occurs when writing a new bank, the file system will sends the new data to an empty bank rather then erasing the existing bank and replacing it. This will cut down on using the erase function improving the life of the hard drive. Also since it only performs writes command and not a erase command and write command like a traditional file system this improves performance of the drive as well.[16]

===Systems Developed For Flash===
In 1999 a Swedish company by the name of Axis Communications, developed and released a file system that was designed specifically to be run on a flash drive. Instead of mapping each physical address space with an emulated block sector, the system creates nodes that store data. The system keeps a log of the nodes and when each node was last updated. The system also keeps track of inodes. These inodes keep a list of nodes that correspond to the relevant data. The nodes also contain information about which inode it belongs to. When the drive is mounted, the system scans all the nodes on the drive, and rebuilds the directory.
As the directory gets built, the nodes containing the data also map the physical location for each piece of data.[14]

When writing data to the drive, the nodes carrying that data get attached to the end of the log.

==Conclusion==

In this way, thanks to its [WHATEVER MAKES LOG FSs ACTUALLY GOOD AT DEALING WITH FLASH], the log-based file-system is far better suited to working with flash memory than a traditional HDD file system. The latter is utterly unfit for this task due to its placing primacy on the minimization of seeks rather than on the minimization and management of erasures. Dealing smartly with erasures is extremely important for a flash memory file system, as that memory type's particular weaknesses, the limited number of erasure cycles, the necessity to erase by the block and the relative slowness of the erasures themselves, all have to do with erasing. A good flash memory file system must therefore be built with the aim of making the best of these weaknesses and this is precisely the reason why older disk-based file systems are not suitable for flash memory while log-based file systems are. [INSPIRATIONAL LAST WORDS]

=Questions=

=References=

[1] Kim, Han-joon; Lee, Sang-goo. ''A New Flash Memory Management for Flash Storage System''. ''IEEExplore''. Dept. of Comput. Sci., Seoul Nat. Univ., 06 Aug 2002. <http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=812717&tag=1#>

[2] Smith, Lance. ''NAND Flash Solid State Storage Performance and Capability''. ''Flash Memory Summit''. SNIA Education Committee, 18 Aug 2009. <http://www.flashmemorysummit.com/English/Collaterals/Proceedings/2009/20090812_T1B_Smith.pdf>

[3] Chang, LiPin. ''On Efficient Wear Leveling for Large-Scale Flash-Memory Storage Systems''. ''Association for Computing Machinery (ACM)''. Dept. of Comput. Sci.,Nat. ChiaoTung Univ., 15 Mar 2007. <http://portal.acm.org/citation.cfm?id=1244248>

[4] Nath, Suman; Gibbons, Phillip. ''Online maintenance of very large random samples on flash storage''. ''Association for Computing Machinery (ACM)''. The VLDB Journal, 27 Jul 2007. <http://portal.acm.org/citation.cfm?id=1731355>

[5] Lim, Seung-Ho; Park; Kyu-Ho. ''An Efficient NAND Flash File System for Flash Memory Storage''. ''CORE Laboratory''. IEEE Transactions On Computers, Jul 2006. <http://vlsi.kaist.ac.kr/paper_list/2006_TC_CFFS.pdf>

[6] ''NAND vs. NOR Flash Memory Technology Overview''. ''RMG and Associates''. Toshiba America, accessed 14 Oct 2010. <http://maltiel-consulting.com/NAND_vs_NOR_Flash_Memory_Technology_Overview_Read_Write_Erase_speed_for_SLC_MLC_semiconductor_consulting_expert.pdf>

[7] Bez, Roberto; Camerlenghi, Emilio; Modelli, Alberto; Visconti, Angelo. ''Introduction to Flash Memory''. ''IEEExplore''. STMicroelectronics, 21 May 2003. <http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1199079&tag=1>

[8] Kawaguchi, Atsuo; Nishioka, Shingo; Motoda Hiroshi. ''A Flash-Memory Based File System''. ''CiteSeerX'' Advanced Research laboratory, Hitachi, Ltd., 1995. <http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.160.5142>

[9] Rosenblum, Mendel; Ouserhout, John. ''The Design and Implementation of a Log-structured File System''. ''Association for Computing Machinery (ACM)''. University of California at Berkeley, Feb 1992. <http://portal.acm.org/citation.cfm?id=146943&coll=GUIDE&dl=GUIDE&CFID=108397378&CFTOKEN=72657973&ret=1#Fulltext>

[10] Shimpi, Anand. ''Intel X25-M SSD: Intel Delivers One of the World's Fastest Drives''. ''AnAndTech''. AnAndTech, 8 Sep 2008. <http://www.anandtech.com/show/2614>

[11] Shimpi, Anand. ''The SSD Relapse: Understanding and Choosing the Best SSD''. ''AnAndTech''. AnAndTech, 30 Aug 2009. <http://www.anandtech.com/show/2829>

[12] Shimpi, Anand. ''The SSD Anthology: Understanding SSDs and New Drives from OCZ''. ''AnAndTech''. AnAndTech, 18 Mar 2009. <http://www.anandtech.com/show/2738>

[13] Corbet, Jonathan. ''Solid-State Storage Devices and the Block Layer''. ''Linux Weekly News''. Linux Weekly News, 4 Oct 2010. <http://lwn.net/Articles/408428/>

[14] Woodhouse, David. ''JFFS : The Journalling Flash File System''. ''CiteSeerX''. Red Hat, Inc, Accessed 14 Oct 2010. <http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.128.6156&rep=rep1&type=pdf>

[15] Agrawal, Nitin; Prabhakaran, Vijayan; Wobber, Ted; Davis, John; Manasse, Mark. Panigrahy, Rina. ''Design Tradeoffs for SSD Performance''. ''Association for Computing Machinery (ACM)'', USENIX 2008 Annual Technical Conference, 2008. <http://portal.acm.org/citation.cfm?id=1404014.1404019>

[16] Lee, Sang-Won, et al. ''A Log Buffer-Based Flash Translation Layer Using Fully-Associative Sector Translation''. ''Association for Computing Machinery (ACM)''. ACM Transactions on Embedded Computing Systems (TECS), Jul 2007. <http://portal.acm.org/citation.cfm?id=1275990>

[17] ''Reach New Heights in Computing Performance''. ''Micron Technology Inc''. Micro Technology Inc, Accessed 14 Oct 2010. <http://www.micron.com/products/solid_state_storage/client_ssd.html>

[18] ''Flash Memories.'' 1 ed. New York: Springer, 1999. Print.

[19] ''Nonvolatile Memory Technologies with Emphasis on Flash: A Comprehensive Guide to Understanding and Using Flash Memory Devices''. ''IEEE Press Series on Microelectronic Systems''. New York: Wiley-Ieee Press, 2008. Print.

[20] ''Nonvolatile Semiconductor Memory Technology: A Comprehensive Guide to Understanding and Using NVSM Devices''. ''IEEE Press Series on Microelectronic Systems''. New York: Wiley-Ieee Press, 1997. Print.

=External links=

Relevant Wikipedia articles: [http://en.wikipedia.org/wiki/Flash_Memory Flash Memory], [http://en.wikipedia.org/wiki/LogFS LogFS], [http://en.wikipedia.org/wiki/Hard_disk Hard Disk Drives], [http://en.wikipedia.org/wiki/Wear_leveling Wear Leveling], [http://en.wikipedia.org/wiki/Hot_spot_%28computer_science%29 Hot Spots], [http://en.wikipedia.org/wiki/Solid-state_drive Sold State Drive].

COMP 3000 Essay 1 2010 Question 10

2010-10-15T07:22:09Z

Gsmith6: /* Banks */

=Question=

How do the constraints of flash storage affect the design of flash-optimized file systems? Explain by contrasting with hard disk-based file systems.

=Answer=
First introduced in the late 80s, Flash-memory is a light, energy-independent, compact, shock-resistant and efficiently readable type of storage. It started out as a replacement for EPROMs. At the time EPROMs needed a UV photoemission to be erased while flash memory could be erased electronically.[7] Because of the particular limitations of this kind of memory, flash file systems require a fundamentally different system architecture than disk-based file-systems: these systems need to be designed in light of flash-memory’s limited number of erase-cycles and its need to conduct erasures one entire block at a time. These constraints are a direct result of the same design that gives flash its advantages with regard to [ TO WHAT?] as both are due to [TO WHAT?] . Thus, a typical disk-based file-system is not suitable for working with flash memory as it erases far too frequently and indiscriminately while being simultaneously optimized for other constraints that do not affect flash memory. This means that a different solution is necessary and that solution is the log-based file-system which is far better suited to working with flash memory because it optimizes erasures by [WHAT?].

==Flash Memory==
Flash memory is non-volatile(meaning digital storage that does not require power to retain its memory) storage space that has become more popular recently due to its fast fetch times. There are two basic forms of the flash storage system, NOR and NAND. Each type has its advantages and disadvantages. NOR has the fastest read times, but is much slower at writing. NAND on the other hand has much more capacity, faster write times, is less expensive, and has a much longer life expectancy.[2]

More and more people use flash memory, with many sizes of drives, ranging from a few hundred megabyte USB key, to a few terabyte internal solid-state drive(SSD). Two main reasons for this movement are because of flash's extremely fast read times, and its falling price. A typical flash drive has read speeds of up to 14 times faster than a hard disk drive (HDD).[17]

This extreme read speed makes flash drives a preferred method of storing games. This effectively makes loading times virtually non-existent. There is however a downside to this method. Games constantly save, modify, and change files wearing out the blocks much quicker. Flash drives have been shown be effective for the use in web-servers for running their CSS scripts or HTML pages.

Although flash drives are exponentially faster than HDDs, they still have not become the main source of data management. The reason for this is because HDDs are simply much cheaper, and flash drives still have many faults. The most critical fault is that each block in flash memory can only be erased approximately 100,000 times.[14] This poses a problem because when modifying a file, even if its a single bit, the entire block must be erased, and rewritten. This erase/rewrite slows down the write operation considerably, making it actually slower to write a file to flash than an HDD.[8]

The transistors that store the data are created with a thin strip of Silicon Oxide separating them. When the erase operation is called on the block where the transistors are located, the system fires electrons down the strip, wiping whatever bits the transistors are holding.

HDDs use a block system, in which the kernel specifies which blocks to read and write. When using a flash drive, the blocks are emulated and mapped to a physical memory address. It does through what is called a "Translational Layer".

==Traditionally Optimized File Systems==

Since the kernel asks for a block number, a conventional hard disk drive (HDD) file-system is not optimized to work with flash memory. The reason for this is that conventional hard-disks have different constraints from those of flash memory - their primary problem is to reduce seeking time, while the primary problem when working with flash memory is to erase in a minimal and balanced way.

The most consuming process for an HDD is seeking data by relocating the read-head and spinning the magnetic disk. A traditional file system optimizes the way it stores data by placing related blocks close-by on the disk in order to minimize mechanical movement within the HDD. One of the great advantages of flash memory, which accounts for its fast read speed, is that there is no need to seek data physically. This is also why defragmentaion, a procedure used by HDDs to put files into more convenient configurations and thus minimize seeking times, loses its purpose in a flash memory context. Indeed, the unnecessary erasures that it entails are both inefficient and harmful for a flash memory unit.

This comes directly out of flash memory's aforementioned constraints: the slow block-sized erasures and the limited number of erase-cycles. Because of these, a flash optimal file system needs to minimize its erase operations and also to spread out its erasures in such a way as to avoid the formation of hot-spots: sections of memory which have undergone a disproportionately high number of erasures and are thus in danger of burning out. This process of spreading out data is referred to as ''wear leveling''. To minimize hotspots, a system using flash memory would have to write new data to empty memory blocks. This method would also call for some sort of garbage collection to conduct necessary erasure operations while the system is idle. It makes better sense to do these at this time because of the slow nature of erasures in the flash memory context. Of course, there is no such feature in a traditional HDD file-system.

==Flash Optimized File Systems==

The process of wear leveling ensures that he drive does not keep erasing and writing to the same block over and over. This is achieved by writing data that doesn’t change often to blocks that have been erased frequently. Wear leveling tries to make all the blocks use up there write cycles at an even pace increasing the overall life of the hard drive.[3] This is achieved through a Log-based File System, often referred to as the Flash Transitional Layer(FTL). Essentially, the drive stores a log that keeps track of how many times each erase sector has been invalidated (or erased). The translational layer has a translation table, where each physical memory address is associated with an emulated block sectors. This allows a traditional file system that uses block sectors to be used on the flash drive. Each block has a flag which keeps track of its state. When a block is being written to, the FTL marks the blocks needed as ''allocated''. This prevents other data being written to the block that has already been allocated. The FTL then goes on to write the data in the allocated blocks. Once it completes the transaction, the system updates the allocated blocks to ''pre-valid''. Once that is completed, the drive marks the invalidated blocks to ''invalid'', while marking the newly written block as ''valid''. This entire flagging process is to ensure that the newly allocated blocks are never mixed up with the invalidated blocks.

===Banks===
The FTL organizes data using structures called banks. When the FTL gets a request to write something to memory, it uses a bank list to determine which area of the drive should be used. Essentially a bank is a group of sequential addresses, that keeps track of when it was last updated using timestamps. The FTL will only write to that bank, and once there is not enough space to write anymore, it switches out the current bank for the one with the most available space. When cleaning up the bank, the system puts it into what is called the Cleaning Bank List and removes it from the Bank List, thus avoiding any chance of some data being written to that bank while something is being erased. [14]

===Cleaner===
When the FTL realizes that there is not enough room to write new data onto the drive, it runs a garbage collection routine. This routine selects a segment to be cleaned, copies all of the valid data into a new segment, then erases everything in the old segment. This frees up the otherwise useless invalidated blocks and by not erasing every block as soon as it becomes invalidated, it saves on the amount of times that the expensive erase operation is called. The kernel can preemptively clean the drive when the system is idle.

===Why a Log File System is efficient for flash had drive===
The file system only writes a bank at a time. This means that the OS can save up the small random writes and write them all at the same time in a bank. This will cut down on the use of the expensive write command improving the overall performance of the hard drive.[9]

If a collision occurs when writing a new bank, the file system will sends the new data to an empty bank rather then erasing the existing bank and replacing it. This will cut down on using the erase function improving the life of the hard drive. Also since it only performs writes command and not a erase command and write command like a traditional file system this improves performance of the drive as well.[16]

===Systems Developed For Flash===
In 1999 a Swedish company by the name of Axis Communications, developed and released a file system that was designed specifically to be run on a flash drive. Instead of mapping each physical address space with an emulated block sector, the system creates nodes that store data. The system keeps a log of the nodes and when each node was last updated. The system also keeps track of inodes. These inodes keep a list of nodes that correspond to the relevant data. The nodes also contain information about which inode it belongs to. When the drive is mounted, the system scans all the nodes on the drive, and rebuilds the directory.
As the directory gets built, the nodes containing the data also map the physical location for each piece of data.[14]

When writing data to the drive, the nodes carrying that data get attached to the end of the log.

==Conclusion==

In this way, thanks to its [WHATEVER MAKES LOG FSs ACTUALLY GOOD AT DEALING WITH FLASH], the log-based file-system is far better suited to working with flash memory than a traditional HDD file system. The latter is utterly unfit for this task due to its placing primacy on the minimization of seeks rather than on the minimization and management of erasures. Dealing smartly with erasures is extremely important for a flash memory file system, as that memory type's particular weaknesses, the limited number of erasure cycles, the necessity to erase by the block and the relative slowness of the erasures themselves, all have to do with erasing. A good flash memory file system must therefore be built with the aim of making the best of these weaknesses and this is precisely the reason why older disk-based file systems are not suitable for flash memory while log-based file systems are. [INSPIRATIONAL LAST WORDS]

=Questions=

=References=

[1] Kim, Han-joon; Lee, Sang-goo. ''A New Flash Memory Management for Flash Storage System''. ''IEEExplore''. Dept. of Comput. Sci., Seoul Nat. Univ., 06 Aug 2002. <http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=812717&tag=1#>

[2] Smith, Lance. ''NAND Flash Solid State Storage Performance and Capability''. ''Flash Memory Summit''. SNIA Education Committee, 18 Aug 2009. <http://www.flashmemorysummit.com/English/Collaterals/Proceedings/2009/20090812_T1B_Smith.pdf>

[3] Chang, LiPin. ''On Efficient Wear Leveling for Large-Scale Flash-Memory Storage Systems''. ''Association for Computing Machinery (ACM)''. Dept. of Comput. Sci.,Nat. ChiaoTung Univ., 15 Mar 2007. <http://portal.acm.org/citation.cfm?id=1244248>

[4] Nath, Suman; Gibbons, Phillip. ''Online maintenance of very large random samples on flash storage''. ''Association for Computing Machinery (ACM)''. The VLDB Journal, 27 Jul 2007. <http://portal.acm.org/citation.cfm?id=1731355>

[5] Lim, Seung-Ho; Park; Kyu-Ho. ''An Efficient NAND Flash File System for Flash Memory Storage''. ''CORE Laboratory''. IEEE Transactions On Computers, Jul 2006. <http://vlsi.kaist.ac.kr/paper_list/2006_TC_CFFS.pdf>

[6] ''NAND vs. NOR Flash Memory Technology Overview''. ''RMG and Associates''. Toshiba America, accessed 14 Oct 2010. <http://maltiel-consulting.com/NAND_vs_NOR_Flash_Memory_Technology_Overview_Read_Write_Erase_speed_for_SLC_MLC_semiconductor_consulting_expert.pdf>

[7] Bez, Roberto; Camerlenghi, Emilio; Modelli, Alberto; Visconti, Angelo. ''Introduction to Flash Memory''. ''IEEExplore''. STMicroelectronics, 21 May 2003. <http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1199079&tag=1>

[8] Kawaguchi, Atsuo; Nishioka, Shingo; Motoda Hiroshi. ''A Flash-Memory Based File System''. ''CiteSeerX'' Advanced Research laboratory, Hitachi, Ltd., 1995. <http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.160.5142>

[9] Rosenblum, Mendel; Ouserhout, John. ''The Design and Implementation of a Log-structured File System''. ''Association for Computing Machinery (ACM)''. University of California at Berkeley, Feb 1992. <http://portal.acm.org/citation.cfm?id=146943&coll=GUIDE&dl=GUIDE&CFID=108397378&CFTOKEN=72657973&ret=1#Fulltext>

[10] Shimpi, Anand. ''Intel X25-M SSD: Intel Delivers One of the World's Fastest Drives''. ''AnAndTech''. AnAndTech, 8 Sep 2008. <http://www.anandtech.com/show/2614>

[11] Shimpi, Anand. ''The SSD Relapse: Understanding and Choosing the Best SSD''. ''AnAndTech''. AnAndTech, 30 Aug 2009. <http://www.anandtech.com/show/2829>

[12] Shimpi, Anand. ''The SSD Anthology: Understanding SSDs and New Drives from OCZ''. ''AnAndTech''. AnAndTech, 18 Mar 2009. <http://www.anandtech.com/show/2738>

[13] Corbet, Jonathan. ''Solid-State Storage Devices and the Block Layer''. ''Linux Weekly News''. Linux Weekly News, 4 Oct 2010. <http://lwn.net/Articles/408428/>

[14] Woodhouse, David. ''JFFS : The Journalling Flash File System''. ''CiteSeerX''. Red Hat, Inc, Accessed 14 Oct 2010. <http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.128.6156&rep=rep1&type=pdf>

[15] Agrawal, Nitin; Prabhakaran, Vijayan; Wobber, Ted; Davis, John; Manasse, Mark. Panigrahy, Rina. ''Design Tradeoffs for SSD Performance''. ''Association for Computing Machinery (ACM)'', USENIX 2008 Annual Technical Conference, 2008. <http://portal.acm.org/citation.cfm?id=1404014.1404019>

[16] Lee, Sang-Won, et al. ''A Log Buffer-Based Flash Translation Layer Using Fully-Associative Sector Translation''. ''Association for Computing Machinery (ACM)''. ACM Transactions on Embedded Computing Systems (TECS), Jul 2007. <http://portal.acm.org/citation.cfm?id=1275990>

[17] ''Reach New Heights in Computing Performance''. ''Micron Technology Inc''. Micro Technology Inc, Accessed 14 Oct 2010. <http://www.micron.com/products/solid_state_storage/client_ssd.html>

[18] ''Flash Memories.'' 1 ed. New York: Springer, 1999. Print.

[19] ''Nonvolatile Memory Technologies with Emphasis on Flash: A Comprehensive Guide to Understanding and Using Flash Memory Devices''. ''IEEE Press Series on Microelectronic Systems''. New York: Wiley-Ieee Press, 2008. Print.

[20] ''Nonvolatile Semiconductor Memory Technology: A Comprehensive Guide to Understanding and Using NVSM Devices''. ''IEEE Press Series on Microelectronic Systems''. New York: Wiley-Ieee Press, 1997. Print.

=External links=

Relevant Wikipedia articles: [http://en.wikipedia.org/wiki/Flash_Memory Flash Memory], [http://en.wikipedia.org/wiki/LogFS LogFS], [http://en.wikipedia.org/wiki/Hard_disk Hard Disk Drives], [http://en.wikipedia.org/wiki/Wear_leveling Wear Leveling], [http://en.wikipedia.org/wiki/Hot_spot_%28computer_science%29 Hot Spots], [http://en.wikipedia.org/wiki/Solid-state_drive Sold State Drive].

COMP 3000 Essay 1 2010 Question 10

2010-10-15T07:20:50Z

Gsmith6: /* Systems Developed For Flash */

=Question=

How do the constraints of flash storage affect the design of flash-optimized file systems? Explain by contrasting with hard disk-based file systems.

=Answer=
First introduced in the late 80s, Flash-memory is a light, energy-independent, compact, shock-resistant and efficiently readable type of storage. It started out as a replacement for EPROMs. At the time EPROMs needed a UV photoemission to be erased while flash memory could be erased electronically.[7] Because of the particular limitations of this kind of memory, flash file systems require a fundamentally different system architecture than disk-based file-systems: these systems need to be designed in light of flash-memory’s limited number of erase-cycles and its need to conduct erasures one entire block at a time. These constraints are a direct result of the same design that gives flash its advantages with regard to [ TO WHAT?] as both are due to [TO WHAT?] . Thus, a typical disk-based file-system is not suitable for working with flash memory as it erases far too frequently and indiscriminately while being simultaneously optimized for other constraints that do not affect flash memory. This means that a different solution is necessary and that solution is the log-based file-system which is far better suited to working with flash memory because it optimizes erasures by [WHAT?].

==Flash Memory==
Flash memory is non-volatile(meaning digital storage that does not require power to retain its memory) storage space that has become more popular recently due to its fast fetch times. There are two basic forms of the flash storage system, NOR and NAND. Each type has its advantages and disadvantages. NOR has the fastest read times, but is much slower at writing. NAND on the other hand has much more capacity, faster write times, is less expensive, and has a much longer life expectancy.[2]

More and more people use flash memory, with many sizes of drives, ranging from a few hundred megabyte USB key, to a few terabyte internal solid-state drive(SSD). Two main reasons for this movement are because of flash's extremely fast read times, and its falling price. A typical flash drive has read speeds of up to 14 times faster than a hard disk drive (HDD).[17]

This extreme read speed makes flash drives a preferred method of storing games. This effectively makes loading times virtually non-existent. There is however a downside to this method. Games constantly save, modify, and change files wearing out the blocks much quicker. Flash drives have been shown be effective for the use in web-servers for running their CSS scripts or HTML pages.

Although flash drives are exponentially faster than HDDs, they still have not become the main source of data management. The reason for this is because HDDs are simply much cheaper, and flash drives still have many faults. The most critical fault is that each block in flash memory can only be erased approximately 100,000 times.[14] This poses a problem because when modifying a file, even if its a single bit, the entire block must be erased, and rewritten. This erase/rewrite slows down the write operation considerably, making it actually slower to write a file to flash than an HDD.[8]

The transistors that store the data are created with a thin strip of Silicon Oxide separating them. When the erase operation is called on the block where the transistors are located, the system fires electrons down the strip, wiping whatever bits the transistors are holding.

HDDs use a block system, in which the kernel specifies which blocks to read and write. When using a flash drive, the blocks are emulated and mapped to a physical memory address. It does through what is called a "Translational Layer".

==Traditionally Optimized File Systems==

Since the kernel asks for a block number, a conventional hard disk drive (HDD) file-system is not optimized to work with flash memory. The reason for this is that conventional hard-disks have different constraints from those of flash memory - their primary problem is to reduce seeking time, while the primary problem when working with flash memory is to erase in a minimal and balanced way.

The most consuming process for an HDD is seeking data by relocating the read-head and spinning the magnetic disk. A traditional file system optimizes the way it stores data by placing related blocks close-by on the disk in order to minimize mechanical movement within the HDD. One of the great advantages of flash memory, which accounts for its fast read speed, is that there is no need to seek data physically. This is also why defragmentaion, a procedure used by HDDs to put files into more convenient configurations and thus minimize seeking times, loses its purpose in a flash memory context. Indeed, the unnecessary erasures that it entails are both inefficient and harmful for a flash memory unit.

This comes directly out of flash memory's aforementioned constraints: the slow block-sized erasures and the limited number of erase-cycles. Because of these, a flash optimal file system needs to minimize its erase operations and also to spread out its erasures in such a way as to avoid the formation of hot-spots: sections of memory which have undergone a disproportionately high number of erasures and are thus in danger of burning out. This process of spreading out data is referred to as ''wear leveling''. To minimize hotspots, a system using flash memory would have to write new data to empty memory blocks. This method would also call for some sort of garbage collection to conduct necessary erasure operations while the system is idle. It makes better sense to do these at this time because of the slow nature of erasures in the flash memory context. Of course, there is no such feature in a traditional HDD file-system.

==Flash Optimized File Systems==

The process of wear leveling ensures that he drive does not keep erasing and writing to the same block over and over. This is achieved by writing data that doesn’t change often to blocks that have been erased frequently. Wear leveling tries to make all the blocks use up there write cycles at an even pace increasing the overall life of the hard drive.[3] This is achieved through a Log-based File System, often referred to as the Flash Transitional Layer(FTL). Essentially, the drive stores a log that keeps track of how many times each erase sector has been invalidated (or erased). The translational layer has a translation table, where each physical memory address is associated with an emulated block sectors. This allows a traditional file system that uses block sectors to be used on the flash drive. Each block has a flag which keeps track of its state. When a block is being written to, the FTL marks the blocks needed as ''allocated''. This prevents other data being written to the block that has already been allocated. The FTL then goes on to write the data in the allocated blocks. Once it completes the transaction, the system updates the allocated blocks to ''pre-valid''. Once that is completed, the drive marks the invalidated blocks to ''invalid'', while marking the newly written block as ''valid''. This entire flagging process is to ensure that the newly allocated blocks are never mixed up with the invalidated blocks.

===Banks===
The FTL organizes data using structures called banks. When the FTL gets a request to write something to memory, it uses a bank list to determine which area of the drive should be used. Essentially a bank is a group of sequential addresses, that keeps track of when it was last updated using timestamps. The FTL will only write to that bank, and once there is not enough space to write anymore, it switches out the current bank for the one with the most available space. When cleaning up the bank, the system puts it into what is called the Cleaning Bank List and removes it from the Bank List, thus avoiding any chance of some data being written to that bank while something is being erased.

===Cleaner===
When the FTL realizes that there is not enough room to write new data onto the drive, it runs a garbage collection routine. This routine selects a segment to be cleaned, copies all of the valid data into a new segment, then erases everything in the old segment. This frees up the otherwise useless invalidated blocks and by not erasing every block as soon as it becomes invalidated, it saves on the amount of times that the expensive erase operation is called. The kernel can preemptively clean the drive when the system is idle.

===Why a Log File System is efficient for flash had drive===
The file system only writes a bank at a time. This means that the OS can save up the small random writes and write them all at the same time in a bank. This will cut down on the use of the expensive write command improving the overall performance of the hard drive.[9]

If a collision occurs when writing a new bank, the file system will sends the new data to an empty bank rather then erasing the existing bank and replacing it. This will cut down on using the erase function improving the life of the hard drive. Also since it only performs writes command and not a erase command and write command like a traditional file system this improves performance of the drive as well.[16]

===Systems Developed For Flash===
In 1999 a Swedish company by the name of Axis Communications, developed and released a file system that was designed specifically to be run on a flash drive. Instead of mapping each physical address space with an emulated block sector, the system creates nodes that store data. The system keeps a log of the nodes and when each node was last updated. The system also keeps track of inodes. These inodes keep a list of nodes that correspond to the relevant data. The nodes also contain information about which inode it belongs to. When the drive is mounted, the system scans all the nodes on the drive, and rebuilds the directory.
As the directory gets built, the nodes containing the data also map the physical location for each piece of data.[14]

When writing data to the drive, the nodes carrying that data get attached to the end of the log.

==Conclusion==

In this way, thanks to its [WHATEVER MAKES LOG FSs ACTUALLY GOOD AT DEALING WITH FLASH], the log-based file-system is far better suited to working with flash memory than a traditional HDD file system. The latter is utterly unfit for this task due to its placing primacy on the minimization of seeks rather than on the minimization and management of erasures. Dealing smartly with erasures is extremely important for a flash memory file system, as that memory type's particular weaknesses, the limited number of erasure cycles, the necessity to erase by the block and the relative slowness of the erasures themselves, all have to do with erasing. A good flash memory file system must therefore be built with the aim of making the best of these weaknesses and this is precisely the reason why older disk-based file systems are not suitable for flash memory while log-based file systems are. [INSPIRATIONAL LAST WORDS]

=Questions=

=References=

[1] Kim, Han-joon; Lee, Sang-goo. ''A New Flash Memory Management for Flash Storage System''. ''IEEExplore''. Dept. of Comput. Sci., Seoul Nat. Univ., 06 Aug 2002. <http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=812717&tag=1#>

[2] Smith, Lance. ''NAND Flash Solid State Storage Performance and Capability''. ''Flash Memory Summit''. SNIA Education Committee, 18 Aug 2009. <http://www.flashmemorysummit.com/English/Collaterals/Proceedings/2009/20090812_T1B_Smith.pdf>

[3] Chang, LiPin. ''On Efficient Wear Leveling for Large-Scale Flash-Memory Storage Systems''. ''Association for Computing Machinery (ACM)''. Dept. of Comput. Sci.,Nat. ChiaoTung Univ., 15 Mar 2007. <http://portal.acm.org/citation.cfm?id=1244248>

[4] Nath, Suman; Gibbons, Phillip. ''Online maintenance of very large random samples on flash storage''. ''Association for Computing Machinery (ACM)''. The VLDB Journal, 27 Jul 2007. <http://portal.acm.org/citation.cfm?id=1731355>

[5] Lim, Seung-Ho; Park; Kyu-Ho. ''An Efficient NAND Flash File System for Flash Memory Storage''. ''CORE Laboratory''. IEEE Transactions On Computers, Jul 2006. <http://vlsi.kaist.ac.kr/paper_list/2006_TC_CFFS.pdf>

[6] ''NAND vs. NOR Flash Memory Technology Overview''. ''RMG and Associates''. Toshiba America, accessed 14 Oct 2010. <http://maltiel-consulting.com/NAND_vs_NOR_Flash_Memory_Technology_Overview_Read_Write_Erase_speed_for_SLC_MLC_semiconductor_consulting_expert.pdf>

[7] Bez, Roberto; Camerlenghi, Emilio; Modelli, Alberto; Visconti, Angelo. ''Introduction to Flash Memory''. ''IEEExplore''. STMicroelectronics, 21 May 2003. <http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1199079&tag=1>

[8] Kawaguchi, Atsuo; Nishioka, Shingo; Motoda Hiroshi. ''A Flash-Memory Based File System''. ''CiteSeerX'' Advanced Research laboratory, Hitachi, Ltd., 1995. <http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.160.5142>

[9] Rosenblum, Mendel; Ouserhout, John. ''The Design and Implementation of a Log-structured File System''. ''Association for Computing Machinery (ACM)''. University of California at Berkeley, Feb 1992. <http://portal.acm.org/citation.cfm?id=146943&coll=GUIDE&dl=GUIDE&CFID=108397378&CFTOKEN=72657973&ret=1#Fulltext>

[10] Shimpi, Anand. ''Intel X25-M SSD: Intel Delivers One of the World's Fastest Drives''. ''AnAndTech''. AnAndTech, 8 Sep 2008. <http://www.anandtech.com/show/2614>

[11] Shimpi, Anand. ''The SSD Relapse: Understanding and Choosing the Best SSD''. ''AnAndTech''. AnAndTech, 30 Aug 2009. <http://www.anandtech.com/show/2829>

[12] Shimpi, Anand. ''The SSD Anthology: Understanding SSDs and New Drives from OCZ''. ''AnAndTech''. AnAndTech, 18 Mar 2009. <http://www.anandtech.com/show/2738>

[13] Corbet, Jonathan. ''Solid-State Storage Devices and the Block Layer''. ''Linux Weekly News''. Linux Weekly News, 4 Oct 2010. <http://lwn.net/Articles/408428/>

[14] Woodhouse, David. ''JFFS : The Journalling Flash File System''. ''CiteSeerX''. Red Hat, Inc, Accessed 14 Oct 2010. <http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.128.6156&rep=rep1&type=pdf>

[15] Agrawal, Nitin; Prabhakaran, Vijayan; Wobber, Ted; Davis, John; Manasse, Mark. Panigrahy, Rina. ''Design Tradeoffs for SSD Performance''. ''Association for Computing Machinery (ACM)'', USENIX 2008 Annual Technical Conference, 2008. <http://portal.acm.org/citation.cfm?id=1404014.1404019>

[16] Lee, Sang-Won, et al. ''A Log Buffer-Based Flash Translation Layer Using Fully-Associative Sector Translation''. ''Association for Computing Machinery (ACM)''. ACM Transactions on Embedded Computing Systems (TECS), Jul 2007. <http://portal.acm.org/citation.cfm?id=1275990>

[17] ''Reach New Heights in Computing Performance''. ''Micron Technology Inc''. Micro Technology Inc, Accessed 14 Oct 2010. <http://www.micron.com/products/solid_state_storage/client_ssd.html>

[18] ''Flash Memories.'' 1 ed. New York: Springer, 1999. Print.

[19] ''Nonvolatile Memory Technologies with Emphasis on Flash: A Comprehensive Guide to Understanding and Using Flash Memory Devices''. ''IEEE Press Series on Microelectronic Systems''. New York: Wiley-Ieee Press, 2008. Print.

[20] ''Nonvolatile Semiconductor Memory Technology: A Comprehensive Guide to Understanding and Using NVSM Devices''. ''IEEE Press Series on Microelectronic Systems''. New York: Wiley-Ieee Press, 1997. Print.

=External links=

Relevant Wikipedia articles: [http://en.wikipedia.org/wiki/Flash_Memory Flash Memory], [http://en.wikipedia.org/wiki/LogFS LogFS], [http://en.wikipedia.org/wiki/Hard_disk Hard Disk Drives], [http://en.wikipedia.org/wiki/Wear_leveling Wear Leveling], [http://en.wikipedia.org/wiki/Hot_spot_%28computer_science%29 Hot Spots], [http://en.wikipedia.org/wiki/Solid-state_drive Sold State Drive].

COMP 3000 Essay 1 2010 Question 10

2010-10-15T07:19:23Z

Gsmith6: /* Systems Developed For Flash */

=Question=

How do the constraints of flash storage affect the design of flash-optimized file systems? Explain by contrasting with hard disk-based file systems.

=Answer=
First introduced in the late 80s, Flash-memory is a light, energy-independent, compact, shock-resistant and efficiently readable type of storage. It started out as a replacement for EPROMs. At the time EPROMs needed a UV photoemission to be erased while flash memory could be erased electronically.[7] Because of the particular limitations of this kind of memory, flash file systems require a fundamentally different system architecture than disk-based file-systems: these systems need to be designed in light of flash-memory’s limited number of erase-cycles and its need to conduct erasures one entire block at a time. These constraints are a direct result of the same design that gives flash its advantages with regard to [ TO WHAT?] as both are due to [TO WHAT?] . Thus, a typical disk-based file-system is not suitable for working with flash memory as it erases far too frequently and indiscriminately while being simultaneously optimized for other constraints that do not affect flash memory. This means that a different solution is necessary and that solution is the log-based file-system which is far better suited to working with flash memory because it optimizes erasures by [WHAT?].

==Flash Memory==
Flash memory is non-volatile(meaning digital storage that does not require power to retain its memory) storage space that has become more popular recently due to its fast fetch times. There are two basic forms of the flash storage system, NOR and NAND. Each type has its advantages and disadvantages. NOR has the fastest read times, but is much slower at writing. NAND on the other hand has much more capacity, faster write times, is less expensive, and has a much longer life expectancy.[2]

More and more people use flash memory, with many sizes of drives, ranging from a few hundred megabyte USB key, to a few terabyte internal solid-state drive(SSD). Two main reasons for this movement are because of flash's extremely fast read times, and its falling price. A typical flash drive has read speeds of up to 14 times faster than a hard disk drive (HDD).[17]

This extreme read speed makes flash drives a preferred method of storing games. This effectively makes loading times virtually non-existent. There is however a downside to this method. Games constantly save, modify, and change files wearing out the blocks much quicker. Flash drives have been shown be effective for the use in web-servers for running their CSS scripts or HTML pages.

Although flash drives are exponentially faster than HDDs, they still have not become the main source of data management. The reason for this is because HDDs are simply much cheaper, and flash drives still have many faults. The most critical fault is that each block in flash memory can only be erased approximately 100,000 times.[14] This poses a problem because when modifying a file, even if its a single bit, the entire block must be erased, and rewritten. This erase/rewrite slows down the write operation considerably, making it actually slower to write a file to flash than an HDD.[8]

The transistors that store the data are created with a thin strip of Silicon Oxide separating them. When the erase operation is called on the block where the transistors are located, the system fires electrons down the strip, wiping whatever bits the transistors are holding.

HDDs use a block system, in which the kernel specifies which blocks to read and write. When using a flash drive, the blocks are emulated and mapped to a physical memory address. It does through what is called a "Translational Layer".

==Traditionally Optimized File Systems==

Since the kernel asks for a block number, a conventional hard disk drive (HDD) file-system is not optimized to work with flash memory. The reason for this is that conventional hard-disks have different constraints from those of flash memory - their primary problem is to reduce seeking time, while the primary problem when working with flash memory is to erase in a minimal and balanced way.

The most consuming process for an HDD is seeking data by relocating the read-head and spinning the magnetic disk. A traditional file system optimizes the way it stores data by placing related blocks close-by on the disk in order to minimize mechanical movement within the HDD. One of the great advantages of flash memory, which accounts for its fast read speed, is that there is no need to seek data physically. This is also why defragmentaion, a procedure used by HDDs to put files into more convenient configurations and thus minimize seeking times, loses its purpose in a flash memory context. Indeed, the unnecessary erasures that it entails are both inefficient and harmful for a flash memory unit.

This comes directly out of flash memory's aforementioned constraints: the slow block-sized erasures and the limited number of erase-cycles. Because of these, a flash optimal file system needs to minimize its erase operations and also to spread out its erasures in such a way as to avoid the formation of hot-spots: sections of memory which have undergone a disproportionately high number of erasures and are thus in danger of burning out. This process of spreading out data is referred to as ''wear leveling''. To minimize hotspots, a system using flash memory would have to write new data to empty memory blocks. This method would also call for some sort of garbage collection to conduct necessary erasure operations while the system is idle. It makes better sense to do these at this time because of the slow nature of erasures in the flash memory context. Of course, there is no such feature in a traditional HDD file-system.

==Flash Optimized File Systems==

The process of wear leveling ensures that he drive does not keep erasing and writing to the same block over and over. This is achieved by writing data that doesn’t change often to blocks that have been erased frequently. Wear leveling tries to make all the blocks use up there write cycles at an even pace increasing the overall life of the hard drive.[3] This is achieved through a Log-based File System, often referred to as the Flash Transitional Layer(FTL). Essentially, the drive stores a log that keeps track of how many times each erase sector has been invalidated (or erased). The translational layer has a translation table, where each physical memory address is associated with an emulated block sectors. This allows a traditional file system that uses block sectors to be used on the flash drive. Each block has a flag which keeps track of its state. When a block is being written to, the FTL marks the blocks needed as ''allocated''. This prevents other data being written to the block that has already been allocated. The FTL then goes on to write the data in the allocated blocks. Once it completes the transaction, the system updates the allocated blocks to ''pre-valid''. Once that is completed, the drive marks the invalidated blocks to ''invalid'', while marking the newly written block as ''valid''. This entire flagging process is to ensure that the newly allocated blocks are never mixed up with the invalidated blocks.

===Banks===
The FTL organizes data using structures called banks. When the FTL gets a request to write something to memory, it uses a bank list to determine which area of the drive should be used. Essentially a bank is a group of sequential addresses, that keeps track of when it was last updated using timestamps. The FTL will only write to that bank, and once there is not enough space to write anymore, it switches out the current bank for the one with the most available space. When cleaning up the bank, the system puts it into what is called the Cleaning Bank List and removes it from the Bank List, thus avoiding any chance of some data being written to that bank while something is being erased.

===Cleaner===
When the FTL realizes that there is not enough room to write new data onto the drive, it runs a garbage collection routine. This routine selects a segment to be cleaned, copies all of the valid data into a new segment, then erases everything in the old segment. This frees up the otherwise useless invalidated blocks and by not erasing every block as soon as it becomes invalidated, it saves on the amount of times that the expensive erase operation is called. The kernel can preemptively clean the drive when the system is idle.

===Why a Log File System is efficient for flash had drive===
The file system only writes a bank at a time. This means that the OS can save up the small random writes and write them all at the same time in a bank. This will cut down on the use of the expensive write command improving the overall performance of the hard drive.[9]

If a collision occurs when writing a new bank, the file system will sends the new data to an empty bank rather then erasing the existing bank and replacing it. This will cut down on using the erase function improving the life of the hard drive. Also since it only performs writes command and not a erase command and write command like a traditional file system this improves performance of the drive as well.[16]

===Systems Developed For Flash===
In 1999 a Swedish company by the name of Axis Communications, developed and released a file system that was designed specifically to be run on a flash drive. Instead of mapping each physical address space with an emulated block sector, the system creates nodes that store data. The system keeps a log of the nodes and when each node was last updated. The system also keeps track of inodes. These inodes keep a list of nodes that correspond to the relevant data. The nodes also contain information about which inode it belongs to. When the drive is mounted, the system scans all the nodes on the drive, and rebuilds the directory.
As the directory gets built, the nodes containing the data also map the physical location for each piece of data.

When writing data to the drive, the nodes carrying that data get attached to the end of the log.

==Conclusion==

In this way, thanks to its [WHATEVER MAKES LOG FSs ACTUALLY GOOD AT DEALING WITH FLASH], the log-based file-system is far better suited to working with flash memory than a traditional HDD file system. The latter is utterly unfit for this task due to its placing primacy on the minimization of seeks rather than on the minimization and management of erasures. Dealing smartly with erasures is extremely important for a flash memory file system, as that memory type's particular weaknesses, the limited number of erasure cycles, the necessity to erase by the block and the relative slowness of the erasures themselves, all have to do with erasing. A good flash memory file system must therefore be built with the aim of making the best of these weaknesses and this is precisely the reason why older disk-based file systems are not suitable for flash memory while log-based file systems are. [INSPIRATIONAL LAST WORDS]

=Questions=

=References=

[1] Kim, Han-joon; Lee, Sang-goo. ''A New Flash Memory Management for Flash Storage System''. ''IEEExplore''. Dept. of Comput. Sci., Seoul Nat. Univ., 06 Aug 2002. <http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=812717&tag=1#>

[2] Smith, Lance. ''NAND Flash Solid State Storage Performance and Capability''. ''Flash Memory Summit''. SNIA Education Committee, 18 Aug 2009. <http://www.flashmemorysummit.com/English/Collaterals/Proceedings/2009/20090812_T1B_Smith.pdf>

[3] Chang, LiPin. ''On Efficient Wear Leveling for Large-Scale Flash-Memory Storage Systems''. ''Association for Computing Machinery (ACM)''. Dept. of Comput. Sci.,Nat. ChiaoTung Univ., 15 Mar 2007. <http://portal.acm.org/citation.cfm?id=1244248>

[4] Nath, Suman; Gibbons, Phillip. ''Online maintenance of very large random samples on flash storage''. ''Association for Computing Machinery (ACM)''. The VLDB Journal, 27 Jul 2007. <http://portal.acm.org/citation.cfm?id=1731355>

[5] Lim, Seung-Ho; Park; Kyu-Ho. ''An Efficient NAND Flash File System for Flash Memory Storage''. ''CORE Laboratory''. IEEE Transactions On Computers, Jul 2006. <http://vlsi.kaist.ac.kr/paper_list/2006_TC_CFFS.pdf>

[6] ''NAND vs. NOR Flash Memory Technology Overview''. ''RMG and Associates''. Toshiba America, accessed 14 Oct 2010. <http://maltiel-consulting.com/NAND_vs_NOR_Flash_Memory_Technology_Overview_Read_Write_Erase_speed_for_SLC_MLC_semiconductor_consulting_expert.pdf>

[7] Bez, Roberto; Camerlenghi, Emilio; Modelli, Alberto; Visconti, Angelo. ''Introduction to Flash Memory''. ''IEEExplore''. STMicroelectronics, 21 May 2003. <http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1199079&tag=1>

[8] Kawaguchi, Atsuo; Nishioka, Shingo; Motoda Hiroshi. ''A Flash-Memory Based File System''. ''CiteSeerX'' Advanced Research laboratory, Hitachi, Ltd., 1995. <http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.160.5142>

[9] Rosenblum, Mendel; Ouserhout, John. ''The Design and Implementation of a Log-structured File System''. ''Association for Computing Machinery (ACM)''. University of California at Berkeley, Feb 1992. <http://portal.acm.org/citation.cfm?id=146943&coll=GUIDE&dl=GUIDE&CFID=108397378&CFTOKEN=72657973&ret=1#Fulltext>

[10] Shimpi, Anand. ''Intel X25-M SSD: Intel Delivers One of the World's Fastest Drives''. ''AnAndTech''. AnAndTech, 8 Sep 2008. <http://www.anandtech.com/show/2614>

[11] Shimpi, Anand. ''The SSD Relapse: Understanding and Choosing the Best SSD''. ''AnAndTech''. AnAndTech, 30 Aug 2009. <http://www.anandtech.com/show/2829>

[12] Shimpi, Anand. ''The SSD Anthology: Understanding SSDs and New Drives from OCZ''. ''AnAndTech''. AnAndTech, 18 Mar 2009. <http://www.anandtech.com/show/2738>

[13] Corbet, Jonathan. ''Solid-State Storage Devices and the Block Layer''. ''Linux Weekly News''. Linux Weekly News, 4 Oct 2010. <http://lwn.net/Articles/408428/>

[14] Woodhouse, David. ''JFFS : The Journalling Flash File System''. ''CiteSeerX''. Red Hat, Inc, Accessed 14 Oct 2010. <http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.128.6156&rep=rep1&type=pdf>

[15] Agrawal, Nitin; Prabhakaran, Vijayan; Wobber, Ted; Davis, John; Manasse, Mark. Panigrahy, Rina. ''Design Tradeoffs for SSD Performance''. ''Association for Computing Machinery (ACM)'', USENIX 2008 Annual Technical Conference, 2008. <http://portal.acm.org/citation.cfm?id=1404014.1404019>

[16] Lee, Sang-Won, et al. ''A Log Buffer-Based Flash Translation Layer Using Fully-Associative Sector Translation''. ''Association for Computing Machinery (ACM)''. ACM Transactions on Embedded Computing Systems (TECS), Jul 2007. <http://portal.acm.org/citation.cfm?id=1275990>

[17] ''Reach New Heights in Computing Performance''. ''Micron Technology Inc''. Micro Technology Inc, Accessed 14 Oct 2010. <http://www.micron.com/products/solid_state_storage/client_ssd.html>

[18] ''Flash Memories.'' 1 ed. New York: Springer, 1999. Print.

[19] ''Nonvolatile Memory Technologies with Emphasis on Flash: A Comprehensive Guide to Understanding and Using Flash Memory Devices''. ''IEEE Press Series on Microelectronic Systems''. New York: Wiley-Ieee Press, 2008. Print.

[20] ''Nonvolatile Semiconductor Memory Technology: A Comprehensive Guide to Understanding and Using NVSM Devices''. ''IEEE Press Series on Microelectronic Systems''. New York: Wiley-Ieee Press, 1997. Print.

=External links=

Relevant Wikipedia articles: [http://en.wikipedia.org/wiki/Flash_Memory Flash Memory], [http://en.wikipedia.org/wiki/LogFS LogFS], [http://en.wikipedia.org/wiki/Hard_disk Hard Disk Drives], [http://en.wikipedia.org/wiki/Wear_leveling Wear Leveling], [http://en.wikipedia.org/wiki/Hot_spot_%28computer_science%29 Hot Spots], [http://en.wikipedia.org/wiki/Solid-state_drive Sold State Drive].

COMP 3000 Essay 1 2010 Question 10

2010-10-15T07:09:10Z

Gsmith6: /* Traditionally Optimized File Systems */

=Question=

How do the constraints of flash storage affect the design of flash-optimized file systems? Explain by contrasting with hard disk-based file systems.

=Answer=
First introduced in the late 80s, Flash-memory is a light, energy-independent, compact, shock-resistant and efficiently readable type of storage. It started out as a replacement for EPROMs. At the time EPROMs needed a UV photoemission to be erased while flash memory could be erased electronically.[7] Because of the particular limitations of this kind of memory, flash file systems require a fundamentally different system architecture than disk-based file-systems: these systems need to be designed in light of flash-memory’s limited number of erase-cycles and its need to conduct erasures one entire block at a time. These constraints are a direct result of the same design that gives flash its advantages with regard to [ TO WHAT?] as both are due to [TO WHAT?] . Thus, a typical disk-based file-system is not suitable for working with flash memory as it erases far too frequently and indiscriminately while being simultaneously optimized for other constraints that do not affect flash memory. This means that a different solution is necessary and that solution is the log-based file-system which is far better suited to working with flash memory because it optimizes erasures by [WHAT?].

==Flash Memory==
Flash memory is non-volatile(meaning digital storage that does not require power to retain its memory) storage space that has become more popular recently due to its fast fetch times. There are two basic forms of the flash storage system, NOR and NAND. Each type has its advantages and disadvantages. NOR has the fastest read times, but is much slower at writing. NAND on the other hand has much more capacity, faster write times, is less expensive, and has a much longer life expectancy.[2]

More and more people use flash memory, with many sizes of drives, ranging from a few hundred megabyte USB key, to a few terabyte internal solid-state drive(SSD). Two main reasons for this movement are because of flash's extremely fast read times, and its falling price. A typical flash drive has read speeds of up to 14 times faster than a hard disk drive (HDD).[17]

This extreme read speed makes flash drives a preferred method of storing games. This effectively makes loading times virtually non-existent. There is however a downside to this method. Games constantly save, modify, and change files wearing out the blocks much quicker. Flash drives have been shown be effective for the use in web-servers for running their CSS scripts or HTML pages.

Although flash drives are exponentially faster than HDDs, they still have not become the main source of data management. The reason for this is because HDDs are simply much cheaper, and flash drives still have many faults. The most critical fault is that each block in flash memory can only be erased approximately 100,000 times.[14] This poses a problem because when modifying a file, even if its a single bit, the entire block must be erased, and rewritten. This erase/rewrite slows down the write operation considerably, making it actually slower to write a file to flash than an HDD.[8]

The transistors that store the data are created with a thin strip of Silicon Oxide separating them. When the erase operation is called on the block where the transistors are located, the system fires electrons down the strip, wiping whatever bits the transistors are holding.

HDDs use a block system, in which the kernel specifies which blocks to read and write. When using a flash drive, the blocks are emulated and mapped to a physical memory address. It does through what is called a "Translational Layer".

==Traditionally Optimized File Systems==

Since the kernel asks for a block number, a conventional hard disk drive (HDD) file-system is not optimized to work with flash memory. The reason for this is that conventional hard-disks have different constraints from those of flash memory - their primary problem is to reduce seeking time, while the primary problem when working with flash memory is to erase in a minimal and balanced way.

The most consuming process for an HDD is seeking data by relocating the read-head and spinning the magnetic disk. A traditional file system optimizes the way it stores data by placing related blocks close-by on the disk in order to minimize mechanical movement within the HDD. One of the great advantages of flash memory, which accounts for its fast read speed, is that there is no need to seek data physically. This is also why defragmentaion, a procedure used by HDDs to put files into more convenient configurations and thus minimize seeking times, loses its purpose in a flash memory context. Indeed, the unnecessary erasures that it entails are both inefficient and harmful for a flash memory unit.

This comes directly out of flash memory's aforementioned constraints: the slow block-sized erasures and the limited number of erase-cycles. Because of these, a flash optimal file system needs to minimize its erase operations and also to spread out its erasures in such a way as to avoid the formation of hot-spots: sections of memory which have undergone a disproportionately high number of erasures and are thus in danger of burning out. This process of spreading out data is referred to as ''wear leveling''. To minimize hotspots, a system using flash memory would have to write new data to empty memory blocks. This method would also call for some sort of garbage collection to conduct necessary erasure operations while the system is idle. It makes better sense to do these at this time because of the slow nature of erasures in the flash memory context. Of course, there is no such feature in a traditional HDD file-system.

==Flash Optimized File Systems==

The process of wear leveling ensures that he drive does not keep erasing and writing to the same block over and over. This is achieved by writing data that doesn’t change often to blocks that have been erased frequently. Wear leveling tries to make all the blocks use up there write cycles at an even pace increasing the overall life of the hard drive.[3] This is achieved through a Log-based File System, often referred to as the Flash Transitional Layer(FTL). Essentially, the drive stores a log that keeps track of how many times each erase sector has been invalidated (or erased). The translational layer has a translation table, where each physical memory address is associated with an emulated block sectors. This allows a traditional file system that uses block sectors to be used on the flash drive. Each block has a flag which keeps track of its state. When a block is being written to, the FTL marks the blocks needed as ''allocated''. This prevents other data being written to the block that has already been allocated. The FTL then goes on to write the data in the allocated blocks. Once it completes the transaction, the system updates the allocated blocks to ''pre-valid''. Once that is completed, the drive marks the invalidated blocks to ''invalid'', while marking the newly written block as ''valid''. This entire flagging process is to ensure that the newly allocated blocks are never mixed up with the invalidated blocks.

===Banks===
The FTL organizes data using structures called banks. When the FTL gets a request to write something to memory, it uses a bank list to determine which area of the drive should be used. Essentially a bank is a group of sequential addresses, that keeps track of when it was last updated using timestamps. The FTL will only write to that bank, and once there is not enough space to write anymore, it switches out the current bank for the one with the most available space. When cleaning up the bank, the system puts it into what is called the Cleaning Bank List and removes it from the Bank List, thus avoiding any chance of some data being written to that bank while something is being erased.

===Cleaner===
When the FTL realizes that there is not enough room to write new data onto the drive, it runs a garbage collection routine. This routine selects a segment to be cleaned, copies all of the valid data into a new segment, then erases everything in the old segment. This frees up the otherwise useless invalidated blocks and by not erasing every block as soon as it becomes invalidated, it saves on the amount of times that the expensive erase operation is called. The kernel can preemptively clean the drive when the system is idle.

===Why a Log File System is efficient for flash had drive===
The file system only writes a bank at a time. This means that the OS can save up the small random writes and write them all at the same time in a bank. This will cut down on the use of the expensive write command improving the overall performance of the hard drive.[9]

If a collision occurs when writing a new bank, the file system will sends the new data to an empty bank rather then erasing the existing bank and replacing it. This will cut down on using the erase function improving the life of the hard drive. Also since it only performs writes command and not a erase command and write command like a traditional file system this improves performance of the drive as well.[16]

===Systems Developed For Flash===
In 1999 a Swedish company by the name of Axis Communications, developed and released a file system that was designed specifically to be run on a flash drive. Instead of mapping each physical address space with an emulated block sector, the system creates nodes that store data. The system also keeps track of inodes. These inodes keep a list of several nodes that correspond to the relevant data.

developed by Axis Communications and first released in 1999
- nodes containing metadata and data are stored sequentially on the device
- each node is associated with a single inode
- starts with header contianing the inode number of the inode to which it belongs and all the current file system metadata for that inode
- node also stores a version number that is written with a version higher than all previous nodes belonging to the same inode
- also stores the inode number and is never reused
- inode metadata includes uid, gid, mtime, atime, mtime, etc
- inode points to several nodes

OPERATION
- the entire drive is scanned when it is mounted
- the raw nodes have the information needed to rebuild the directory and a map for each inode of the phyical location
- when writing files, a node holding the data is attached to the end of the log

==Conclusion==

In this way, thanks to its [WHATEVER MAKES LOG FSs ACTUALLY GOOD AT DEALING WITH FLASH], the log-based file-system is far better suited to working with flash memory than a traditional HDD file system. The latter is utterly unfit for this task due to its placing primacy on the minimization of seeks rather than on the minimization and management of erasures. Dealing smartly with erasures is extremely important for a flash memory file system, as that memory type's particular weaknesses, the limited number of erasure cycles, the necessity to erase by the block and the relative slowness of the erasures themselves, all have to do with erasing. A good flash memory file system must therefore be built with the aim of making the best of these weaknesses and this is precisely the reason why older disk-based file systems are not suitable for flash memory while log-based file systems are. [INSPIRATIONAL LAST WORDS]

=Questions=

=References=

[1] Kim, Han-joon; Lee, Sang-goo. ''A New Flash Memory Management for Flash Storage System''. ''IEEExplore''. Dept. of Comput. Sci., Seoul Nat. Univ., 06 Aug 2002. <http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=812717&tag=1#>

[2] Smith, Lance. ''NAND Flash Solid State Storage Performance and Capability''. ''Flash Memory Summit''. SNIA Education Committee, 18 Aug 2009. <http://www.flashmemorysummit.com/English/Collaterals/Proceedings/2009/20090812_T1B_Smith.pdf>

[3] Chang, LiPin. ''On Efficient Wear Leveling for Large-Scale Flash-Memory Storage Systems''. ''Association for Computing Machinery (ACM)''. Dept. of Comput. Sci.,Nat. ChiaoTung Univ., 15 Mar 2007. <http://portal.acm.org/citation.cfm?id=1244248>

[4] Nath, Suman; Gibbons, Phillip. ''Online maintenance of very large random samples on flash storage''. ''Association for Computing Machinery (ACM)''. The VLDB Journal, 27 Jul 2007. <http://portal.acm.org/citation.cfm?id=1731355>

[5] Lim, Seung-Ho; Park; Kyu-Ho. ''An Efficient NAND Flash File System for Flash Memory Storage''. ''CORE Laboratory''. IEEE Transactions On Computers, Jul 2006. <http://vlsi.kaist.ac.kr/paper_list/2006_TC_CFFS.pdf>

[6] ''NAND vs. NOR Flash Memory Technology Overview''. ''RMG and Associates''. Toshiba America, accessed 14 Oct 2010. <http://maltiel-consulting.com/NAND_vs_NOR_Flash_Memory_Technology_Overview_Read_Write_Erase_speed_for_SLC_MLC_semiconductor_consulting_expert.pdf>

[7] Bez, Roberto; Camerlenghi, Emilio; Modelli, Alberto; Visconti, Angelo. ''Introduction to Flash Memory''. ''IEEExplore''. STMicroelectronics, 21 May 2003. <http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1199079&tag=1>

[8] Kawaguchi, Atsuo; Nishioka, Shingo; Motoda Hiroshi. ''A Flash-Memory Based File System''. ''CiteSeerX'' Advanced Research laboratory, Hitachi, Ltd., 1995. <http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.160.5142>

[9] Rosenblum, Mendel; Ouserhout, John. ''The Design and Implementation of a Log-structured File System''. ''Association for Computing Machinery (ACM)''. University of California at Berkeley, Feb 1992. <http://portal.acm.org/citation.cfm?id=146943&coll=GUIDE&dl=GUIDE&CFID=108397378&CFTOKEN=72657973&ret=1#Fulltext>

[10] Shimpi, Anand. ''Intel X25-M SSD: Intel Delivers One of the World's Fastest Drives''. ''AnAndTech''. AnAndTech, 8 Sep 2008. <http://www.anandtech.com/show/2614>

[11] Shimpi, Anand. ''The SSD Relapse: Understanding and Choosing the Best SSD''. ''AnAndTech''. AnAndTech, 30 Aug 2009. <http://www.anandtech.com/show/2829>

[12] Shimpi, Anand. ''The SSD Anthology: Understanding SSDs and New Drives from OCZ''. ''AnAndTech''. AnAndTech, 18 Mar 2009. <http://www.anandtech.com/show/2738>

[13] Corbet, Jonathan. ''Solid-State Storage Devices and the Block Layer''. ''Linux Weekly News''. Linux Weekly News, 4 Oct 2010. <http://lwn.net/Articles/408428/>

[14] Woodhouse, David. ''JFFS : The Journalling Flash File System''. ''CiteSeerX''. Red Hat, Inc, Accessed 14 Oct 2010. <http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.128.6156&rep=rep1&type=pdf>

[15] Agrawal, Nitin; Prabhakaran, Vijayan; Wobber, Ted; Davis, John; Manasse, Mark. Panigrahy, Rina. ''Design Tradeoffs for SSD Performance''. ''Association for Computing Machinery (ACM)'', USENIX 2008 Annual Technical Conference, 2008. <http://portal.acm.org/citation.cfm?id=1404014.1404019>

[16] Lee, Sang-Won, et al. ''A Log Buffer-Based Flash Translation Layer Using Fully-Associative Sector Translation''. ''Association for Computing Machinery (ACM)''. ACM Transactions on Embedded Computing Systems (TECS), Jul 2007. <http://portal.acm.org/citation.cfm?id=1275990>

[17] ''Reach New Heights in Computing Performance''. ''Micron Technology Inc''. Micro Technology Inc, Accessed 14 Oct 2010. <http://www.micron.com/products/solid_state_storage/client_ssd.html>

[18] ''Flash Memories.'' 1 ed. New York: Springer, 1999. Print.

[19] ''Nonvolatile Memory Technologies with Emphasis on Flash: A Comprehensive Guide to Understanding and Using Flash Memory Devices''. ''IEEE Press Series on Microelectronic Systems''. New York: Wiley-Ieee Press, 2008. Print.

[20] ''Nonvolatile Semiconductor Memory Technology: A Comprehensive Guide to Understanding and Using NVSM Devices''. ''IEEE Press Series on Microelectronic Systems''. New York: Wiley-Ieee Press, 1997. Print.

=External links=

Relevant Wikipedia articles: [http://en.wikipedia.org/wiki/Flash_Memory Flash Memory], [http://en.wikipedia.org/wiki/LogFS LogFS], [http://en.wikipedia.org/wiki/Hard_disk Hard Disk Drives], [http://en.wikipedia.org/wiki/Wear_leveling Wear Leveling], [http://en.wikipedia.org/wiki/Hot_spot_%28computer_science%29 Hot Spots], [http://en.wikipedia.org/wiki/Solid-state_drive Sold State Drive].

COMP 3000 Essay 1 2010 Question 10

2010-10-15T06:28:26Z

Gsmith6: /* Answer */

=Question=

How do the constraints of flash storage affect the design of flash-optimized file systems? Explain by contrasting with hard disk-based file systems.

=Answer=
First introduced in the late 80s, Flash-memory is a light, energy-independent, compact, shock-resistant and efficiently readable type of storage. It started out as a replacement for EPROMs. At the time EPROMs needed a UV photoemission to be erased while flash memory could be erased electronically.[7] Because of the particular limitations of this kind of memory, flash file systems require a fundamentally different system architecture than disk-based file-systems: these systems need to be designed in light of flash-memory’s limited number of erase-cycles and its need to conduct erasures one entire block at a time. These constraints are a direct result of the same design that gives flash its advantages with regard to [ TO WHAT?] as both are due to [TO WHAT?] . Thus, a typical disk-based file-system is not suitable for working with flash memory as it erases far too frequently and indiscriminately while being simultaneously optimized for other constraints that do not affect flash memory. This means that a different solution is necessary and that solution is the log-based file-system which is far better suited to working with flash memory because it optimizes erasures by [WHAT?].

==Flash Memory==
Flash memory is non-volatile(meaning digital storage that does not require power to retain its memory) storage space that has become more popular recently due to its fast fetch times. There are two basic forms of the flash storage system, NOR and NAND. Each type has its advantages and disadvantages. NOR has the fastest read times, but is much slower at writing. NAND on the other hand has much more capacity, faster write times, is less expensive, and has a much longer life expectancy.[2]

More and more people use flash memory, with many sizes of drives, ranging from a few hundred megabyte USB key, to a few terabyte internal solid-state drive(SSD). Two main reasons for this movement are because of flash's extremely fast read times, and its falling price. A typical flash drive has read speeds of up to 14 times faster than a hard disk drive (HDD).[17]

This extreme read speed makes flash drives a preferred method of storing games. This effectively makes loading times virtually non-existent. There is however a downside to this method. Games constantly save, modify, and change files wearing out the blocks much quicker. Flash drives have been shown be effective for the use in web-servers for running their CSS scripts or HTML pages.

Although flash drives are exponentially faster than HDDs, they still have not become the main source of data management. The reason for this is because HDDs are simply much cheaper, and flash drives still have many faults. The most critical fault is that each block in flash memory can only be erased approximately 100,000 times.[14] This poses a problem because when modifying a file, even if its a single bit, the entire block must be erased, and rewritten. This erase/rewrite slows down the write operation considerably, making it actually slower to write a file to flash than an HDD.[8]

The transistors that store the data are created with a thin strip of Silicon Oxide separating them. When the erase operation is called on the block where the transistors are located, the system fires electrons down the strip, wiping whatever bits the transistors are holding.

HDDs use a block system, in which the kernel specifies which blocks to read and write. When using a flash drive, the blocks are emulated and mapped to a physical memory address. It does through what is called a "Translational Layer".

==Traditionally Optimized File Systems==

Since the kernel asks for a block number, a conventional hard disk drive (HDD) file-system is not optimized to work with flash memory. The reason for this is that conventional hard-disks have different constraints from those of flash memory - their primary problem is to reduce seeking time, while the primary problem when working with flash memory is to erase in a minimal and balanced way.

The most consuming process for an HDD is seeking data by relocating the read-head and spinning the magnetic disk. A traditional file system optimizes the way it stores data by placing related blocks close-by on the disk in order to minimize mechanical movement within the HDD. One of the great advantages of flash memory, which accounts for its fast read speed, is that there is no need to seek data physically. This is also why defragmentaion, a procedure used by HDDs to put files into more convenient configurations and thus minimize seeking times, loses its purpose in a flash memory context. Indeed, the unnecessary erasures that it entails are both inefficient and harmful for a flash memory unit.

This comes directly out of flash memory's aforementioned constraints: the slow block-sized erasures and the limited number of erase-cycles. Because of these, a flash optimal file system needs to minimize its erase operations and also to spread out its erasures in such a way as to avoid the formation of hot-spots: sections of memory which have undergone a disproportionately high number of erasures and are thus in danger of burning out. This process of spreading out data is referred to as "wear leveling". To minimize hotspots, a system using flash memory would have to write new data to empty memory blocks. This method would also call for some sort of garbage collection to conduct necessary erasure operations while the system is idle. It makes better sense to do these at this time because of the slow nature of erasures in the flash memory context. Of course, there is no such feature in a traditional HDD file-system.

'''More on this later'''

==Flash Optimized File Systems==

The process of wear leveling ensures that he drive does not keep erasing and writing to the same block over and over. This is achieved by writing data that doesn’t change often to blocks that have been erased frequently. Wear leveling tries to make all the blocks use up there write cycles at an even pace increasing the overall life of the hard drive.[3] This is achieved through a Log-based File System, often referred to as the Flash Transitional Layer(FTL). Essentially, the drive stores a log that keeps track of how many times each erase sector has been invalidated (or erased). The translational layer has a translation table, where each physical memory address is associated with an emulated block sectors. This allows a traditional file system that uses block sectors to be used on the flash drive. Each block has a flag which keeps track of its state. When a block is being written to, the FTL marks the blocks needed as ''allocated''. This prevents other data being written to the block that has already been allocated. The FTL then goes on to write the data in the allocated blocks. Once it completes the transaction, the system updates the allocated blocks to ''pre-valid''. Once that is completed, the drive marks the invalidated blocks to ''invalid'', while marking the newly written block as ''valid''. This entire flagging process is to ensure that the newly allocated blocks are never mixed up with the invalidated blocks.

===Banks===
The FTL organizes data using structures called banks. When the FTL gets a request to write something to memory, it uses a bank list to determine which area of the drive should be used. Essentially a bank is a group of sequential addresses, that keeps track of when it was last updated using timestamps. The FTL will only write to that bank, and once there is not enough space to write anymore, it switches out the current bank for the one with the most available space. When cleaning up the bank, the system puts it into what is called the Cleaning Bank List and removes it from the Bank List, thus avoiding any chance of some data being written to that bank while something is being erased.

===Cleaner===
When the FTL realizes that there is not enough room to write new data onto the drive, it runs a garbage collection routine. This routine selects a segment to be cleaned, copies all of the valid data into a new segment, then erases everything in the old segment. This frees up the otherwise useless invalidated blocks and by not erasing every block as soon as it becomes invalidated, it saves on the amount of times that the expensive erase operation is called. The kernel can preemptively clean the drive when the system is idle.

===Systems Developed For Flash===
In 1999 a Swedish company by the name of Axis Communications, developed and released a file system that was designed specifically to be run on a flash drive. Instead of mapping each physical address space with an emulated block sector, the system creates nodes that store data. The system also keeps track of inodes. These inodes keep a list of several nodes that correspond to the relevant data.

developed by Axis Communications and first released in 1999
- nodes containing metadata and data are stored sequentially on the device
- each node is associated with a single inode
- starts with header contianing the inode number of the inode to which it belongs and all the current file system metadata for that inode
- node also stores a version number that is written with a version higher than all previous nodes belonging to the same inode
- also stores the inode number and is never reused
- inode metadata includes uid, gid, mtime, atime, mtime, etc
- inode points to several nodes

OPERATION
- the entire drive is scanned when it is mounted
- the raw nodes have the information needed to rebuild the directory and a map for each inode of the phyical location
- when writing files, a node holding the data is attached to the end of the log

==Conclusion==

In this way, thanks to its [WHATEVER MAKES LOG FSs ACTUALLY GOOD AT DEALING WITH FLASH], the log-based file-system is far better suited to working with flash memory than a traditional HDD file system. The latter is utterly unfit for this task due to its placing primacy on the minimization of seeks rather than on the minimization and management of erasures. Dealing smartly with erasures is extremely important for a flash memory file system, as that memory type's particular weaknesses, the limited number of erasure cycles, the necessity to erase by the block and the relative slowness of the erasures themselves, all have to do with erasing. A good flash memory file system must therefore be built with the aim of making the best of these weaknesses and this is precisely the reason why older disk-based file systems are not suitable for flash memory while log-based file systems are. [INSPIRATIONAL LAST WORDS]

=Questions=

=References=

[1] Kim, Han-joon; Lee, Sang-goo. ''A New Flash Memory Management for Flash Storage System''. ''IEEExplore''. Dept. of Comput. Sci., Seoul Nat. Univ., 06 Aug 2002. <http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=812717&tag=1#>

[2] Smith, Lance. ''NAND Flash Solid State Storage Performance and Capability''. ''Flash Memory Summit''. SNIA Education Committee, 18 Aug 2009. <http://www.flashmemorysummit.com/English/Collaterals/Proceedings/2009/20090812_T1B_Smith.pdf>

[3] Chang, LiPin. ''On Efficient Wear Leveling for Large-Scale Flash-Memory Storage Systems''. ''Association for Computing Machinery (ACM)''. Dept. of Comput. Sci.,Nat. ChiaoTung Univ., 15 Mar 2007. <http://portal.acm.org/citation.cfm?id=1244248>

[4] Nath, Suman; Gibbons, Phillip. ''Online maintenance of very large random samples on flash storage''. ''Association for Computing Machinery (ACM)''. The VLDB Journal, 27 Jul 2007. <http://portal.acm.org/citation.cfm?id=1731355>

[5] Lim, Seung-Ho; Park; Kyu-Ho. ''An Efficient NAND Flash File System for Flash Memory Storage''. ''CORE Laboratory''. IEEE Transactions On Computers, Jul 2006. <http://vlsi.kaist.ac.kr/paper_list/2006_TC_CFFS.pdf>

[6] ''NAND vs. NOR Flash Memory Technology Overview''. ''RMG and Associates''. Toshiba America, accessed 14 Oct 2010. <http://maltiel-consulting.com/NAND_vs_NOR_Flash_Memory_Technology_Overview_Read_Write_Erase_speed_for_SLC_MLC_semiconductor_consulting_expert.pdf>

[7] Bez, Roberto; Camerlenghi, Emilio; Modelli, Alberto; Visconti, Angelo. ''Introduction to Flash Memory''. ''IEEExplore''. STMicroelectronics, 21 May 2003. <http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1199079&tag=1>

[8] Kawaguchi, Atsuo; Nishioka, Shingo; Motoda Hiroshi. ''A Flash-Memory Based File System''. ''CiteSeerX'' Advanced Research laboratory, Hitachi, Ltd., 1995. <http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.160.5142>

[9] Rosenblum, Mendel; Ouserhout, John. ''The Design and Implementation of a Log-structured File System''. ''Association for Computing Machinery (ACM)''. University of California at Berkeley, Feb 1992. <http://portal.acm.org/citation.cfm?id=146943&coll=GUIDE&dl=GUIDE&CFID=108397378&CFTOKEN=72657973&ret=1#Fulltext>

[10] Shimpi, Anand. ''Intel X25-M SSD: Intel Delivers One of the World's Fastest Drives''. ''AnAndTech''. AnAndTech, 8 Sep 2008. <http://www.anandtech.com/show/2614>

[11] Shimpi, Anand. ''The SSD Relapse: Understanding and Choosing the Best SSD''. ''AnAndTech''. AnAndTech, 30 Aug 2009. <http://www.anandtech.com/show/2829>

[12] Shimpi, Anand. ''The SSD Anthology: Understanding SSDs and New Drives from OCZ''. ''AnAndTech''. AnAndTech, 18 Mar 2009. <http://www.anandtech.com/show/2738>

[13] Corbet, Jonathan. ''Solid-State Storage Devices and the Block Layer''. ''Linux Weekly News''. Linux Weekly News, 4 Oct 2010. <http://lwn.net/Articles/408428/>

[14] Woodhouse, David. ''JFFS : The Journalling Flash File System''. ''CiteSeerX''. Red Hat, Inc, Accessed 14 Oct 2010. <http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.128.6156&rep=rep1&type=pdf>

[15] Agrawal, Nitin; Prabhakaran, Vijayan; Wobber, Ted; Davis, John; Manasse, Mark. Panigrahy, Rina. ''Design Tradeoffs for SSD Performance''. ''Association for Computing Machinery (ACM)'', USENIX 2008 Annual Technical Conference, 2008. <http://portal.acm.org/citation.cfm?id=1404014.1404019>

[16] Lee, Sang-Won, et al. ''A Log Buffer-Based Flash Translation Layer Using Fully-Associative Sector Translation''. ''Association for Computing Machinery (ACM)''. ACM Transactions on Embedded Computing Systems (TECS), Jul 2007. <http://portal.acm.org/citation.cfm?id=1275990>

[17] ''Reach New Heights in Computing Performance''. ''Micron Technology Inc''. Micro Technology Inc, Accessed 14 Oct 2010. <http://www.micron.com/products/solid_state_storage/client_ssd.html>

[18] ''Flash Memories.'' 1 ed. New York: Springer, 1999. Print.

[19] ''Nonvolatile Memory Technologies with Emphasis on Flash: A Comprehensive Guide to Understanding and Using Flash Memory Devices''. ''IEEE Press Series on Microelectronic Systems''. New York: Wiley-Ieee Press, 2008. Print.

[20] ''Nonvolatile Semiconductor Memory Technology: A Comprehensive Guide to Understanding and Using NVSM Devices''. ''IEEE Press Series on Microelectronic Systems''. New York: Wiley-Ieee Press, 1997. Print.

=External links=

Relevant Wikipedia articles: [http://en.wikipedia.org/wiki/Flash_Memory Flash Memory], [http://en.wikipedia.org/wiki/LogFS LogFS], [http://en.wikipedia.org/wiki/Hard_disk Hard Disk Drives], [http://en.wikipedia.org/wiki/Wear_leveling Wear Leveling], [http://en.wikipedia.org/wiki/Hot_spot_%28computer_science%29 Hot Spots], [http://en.wikipedia.org/wiki/Solid-state_drive Sold State Drive].

COMP 3000 Essay 1 2010 Question 10

2010-10-15T05:52:28Z

Gsmith6: /* Answer */

=Question=

How do the constraints of flash storage affect the design of flash-optimized file systems? Explain by contrasting with hard disk-based file systems.

=Answer=
First introduced in the late 80s, Flash-memory is a light, energy-independent, compact, shock-resistant and efficiently readable type of storage. It started out as a replacement for EPROMs. At the time EPROMs needed a UV photoemission to be erased while flash memory could be erased electronically.[7] Because of the particular limitations of this kind of memory, flash file systems require a fundamentally different system architecture than disk-based file-systems: these systems need to be designed in light of flash-memory’s limited number of erase-cycles and its need to conduct erasures one entire block at a time. These constraints are a direct result of the same design that gives flash its advantages with regard to [ TO WHAT?] as both are due to [TO WHAT?] . Thus, a typical disk-based file-system is not suitable for working with flash memory as it erases far too frequently and indiscriminately while being simultaneously optimized for other constraints that do not affect flash memory. This means that a different solution is necessary and that solution is the log-based file-system which is far better suited to working with flash memory because it optimizes erasures by [WHAT?].

==Flash Memory==
Flash memory is non-volatile(meaning digital storage that does not require power to retain its memory) storage space that has become more popular recently due to its fast fetch times. There are two basic forms of the flash storage system, NOR and NAND. Each type has its advantages and disadvantages. NOR has the fastest read times, but is much slower at writing. NAND on the other hand has much more capacity, faster write times, is less expensive, and has a much longer life expectancy.[2]

Flash memory started out as a replacement for EPROMs. At the time EPROMs needed a UV photoemission to be erased while flash memory could be erased electronically. The first flash memory product came out in 1988 but it did not take off until the late 1990’s because it could not be reliable produced. NOR and NAND memory is named after the arrangement of the cells in the memory array. NOR based flash memory benefits from having very fast burst read times but slower write times. Due to the structure of NOR memory programs stored in NOR based memory can be executed without being loaded into RAM first. NAND flash memory has a very large storage capacity and can read and write large files relatively fast. NAND is more suited for storage while NOR memory is better suited for direct program execution such as in CMOS chips '''-not sure if this is rephrased'''

More and more people use flash memory, with many sizes of drives, ranging from a few hundred megabyte USB key, to a few terabyte internal solid-state drive(SSD). Two main reasons for this movement are because of flash's extremely fast read times, and its falling price. A typical flash drive has read speeds of up to 14 times faster than a hard disk drive (HDD).[17]

This extreme read speed makes flash drives a preferred method of storing games. This effectively makes loading times virtually non-existent. There is however a downside to this method. Games constantly save, modify, and change files wearing out the blocks much quicker. Flash drives have been shown be effective for the use in web-servers for running their CSS scripts or HTML pages.

Although flash drives are exponentially faster than HDDs, they still have not become the main source of data management. The reason for this is because HDDs are simply much cheaper, and flash drives still have many faults. The most critical fault is that each block in flash memory can only be erased approximately 100,000 times.[14] This poses a problem because when modifying a file, even if its a single bit, the entire block must be erased, and rewritten. This erase/rewrite slows down the write operation considerably, making it actually slower to write a file to flash than an HDD.[8]

The transistors that store the data are created with a thin strip of Silicon Oxide separating them. When the erase operation is called on the block where the transistors are located, the system fires electrons down the strip, wiping whatever bits the transistors are holding.

HDDs use a block system, in which the kernel specifies which blocks to read and write. When using a flash drive, the blocks are emulated and mapped to a physical memory address. It does through what is called a "Translational Layer".

==Traditionally Optimized File Systems==

Since the kernel asks for a block number, a conventional hard disk drive (HDD) file-system is not optimized to work with flash memory. The reason for this is that conventional hard-disks have different constraints from those of flash memory - their primary problem is to reduce seeking time, while the primary problem when working with flash memory is to erase in a minimal and balanced way.

The most consuming process for an HDD is seeking data by relocating the read-head and spinning the magnetic disk. A traditional file system optimizes the way it stores data by placing related blocks close-by on the disk in order to minimize mechanical movement within the HDD. One of the great advantages of flash memory, which accounts for its fast read speed, is that there is no need to seek data physically. This is also why defragmentaion, a procedure used by HDDs to put files into more convenient configurations and thus minimize seeking times, loses its purpose in a flash memory context. Indeed, the unnecessary erasures that it entails are both inefficient and harmful for a flash memory unit.

This comes directly out of flash memory's aforementioned constraints: the slow block-sized erasures and the limited number of erase-cycles. Because of these, a flash optimal file system needs to minimize its erase operations and also to spread out its erasures in such a way as to avoid the formation of hot-spots: sections of memory which have undergone a disproportionately high number of erasures and are thus in danger of burning out. This process of spreading out data is referred to as "wear leveling". To minimize hotspots, a system using flash memory would have to write new data to empty memory blocks. This method would also call for some sort of garbage collection to conduct necessary erasure operations while the system is idle. It makes better sense to do these at this time because of the slow nature of erasures in the flash memory context. Of course, there is no such feature in a traditional HDD file-system.

'''More on this later'''

==Flash Optimized File Systems==

The process of "wear leveling" ensures that he drive does not keep erasing and writing to the same block over and over. This is achieved by writing data that doesn’t change often to blocks that have been erased frequently. Wear leveling tries to make all the blocks use up there write cycles at an even pace increasing the overall life of the hard drive.[3] This is achieved through a Log-based File System, often referred to as the Flash Transitional Layer(FTL). Essentially, the drive stores a log that keeps track of how many times each erase sector has been invalidated (or erased). The translational layer has a translation table, where each physical memory address is associated with an emulated block sectors. This allows a traditional file system that uses block sectors to be used on the flash drive. Each block has a flag which keeps track of its state. When a block is being written to, the FTL marks the blocks needed as ''allocated''. This prevents other data being written to the block that has already been allocated. The FTL then goes on to write the data in the allocated blocks. Once it completes the transaction, the system updates the allocated blocks to ''pre-valid''. Once that is completed, the drive marks the invalidated blocks to ''invalid'', while marking the newly written block as ''valid''. This entire flagging process is to ensure that the newly allocated blocks are never mixed up with the invalidated blocks.

===Banks===
The FTL organizes data using structures called banks. When the FTL gets a request to write something to memory, it uses a bank list to determine which area of the drive should be used. Essentially a bank is a group of sequential addresses, that keeps track of when it was last updated using timestamps. The FTL will only write to that bank, and once there is not enough space to write anymore, it switches out the current bank for the one with the most available space. When cleaning up the bank, the system puts it into what is called the Cleaning Bank List and removes it from the Bank List, thus avoiding any chance of some data being written to that bank while something is being erased.

===Cleaner===
When the FTL realizes that there is not enough room to write new data onto the drive, it runs a garbage collection routine. This routine selects a segment to be cleaned, copies all of the valid data into a new segment, then erases everything in the old segment. This frees up the otherwise useless invalidated blocks and by not erasing every block as soon as it becomes invalidated, it saves on the amount of times that the expensive erase operation is called. The kernel can preemptively clean the drive when the system is idle.

==Conclusion==

In this way, thanks to its [WHATEVER MAKES LOG FSs ACTUALLY GOOD AT DEALING WITH FLASH], the log-based file-system is far better suited to working with flash memory than a traditional HDD file system. The latter is utterly unfit for this task due to its placing primacy on the minimization of seeks rather than on the minimization and management of erasures. Dealing smartly with erasures is extremely important for a flash memory file system, as that memory type's particular weaknesses, the limited number of erasure cycles, the necessity to erase by the block and the relative slowness of the erasures themselves, all have to do with erasing. A good flash memory file system must therefore be built with the aim of making the best of these weaknesses and this is precisely the reason why older disk-based file systems are not suitable for flash memory while log-based file systems are. [INSPIRATIONAL LAST WORDS]

=Questions=

=References=

[1] Kim, Han-joon; Lee, Sang-goo. ''A New Flash Memory Management for Flash Storage System''. ''IEEExplore''. Dept. of Comput. Sci., Seoul Nat. Univ., 06 Aug 2002. <http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=812717&tag=1#>

[2] Smith, Lance. ''NAND Flash Solid State Storage Performance and Capability''. ''Flash Memory Summit''. SNIA Education Committee, 18 Aug 2009. <http://www.flashmemorysummit.com/English/Collaterals/Proceedings/2009/20090812_T1B_Smith.pdf>

[3] Chang, LiPin. ''On Efficient Wear Leveling for Large-Scale Flash-Memory Storage Systems''. ''Association for Computing Machinery (ACM)''. Dept. of Comput. Sci.,Nat. ChiaoTung Univ., 15 Mar 2007. <http://portal.acm.org/citation.cfm?id=1244248>

[4] Nath, Suman; Gibbons, Phillip. ''Online maintenance of very large random samples on flash storage''. ''Association for Computing Machinery (ACM)''. The VLDB Journal, 27 Jul 2007. <http://portal.acm.org/citation.cfm?id=1731355>

[5] Lim, Seung-Ho; Park; Kyu-Ho. ''An Efficient NAND Flash File System for Flash Memory Storage''. ''CORE Laboratory''. IEEE Transactions On Computers, Jul 2006. <http://vlsi.kaist.ac.kr/paper_list/2006_TC_CFFS.pdf>

[6] ''NAND vs. NOR Flash Memory Technology Overview''. ''RMG and Associates''. Toshiba America, accessed 14 Oct 2010. <http://maltiel-consulting.com/NAND_vs_NOR_Flash_Memory_Technology_Overview_Read_Write_Erase_speed_for_SLC_MLC_semiconductor_consulting_expert.pdf>

[7] Bez, Roberto; Camerlenghi, Emilio; Modelli, Alberto; Visconti, Angelo. ''Introduction to Flash Memory''. ''IEEExplore''. STMicroelectronics, 21 May 2003. <http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1199079&tag=1>

[8] Kawaguchi, Atsuo; Nishioka, Shingo; Motoda Hiroshi. ''A Flash-Memory Based File System''. ''CiteSeerX'' Advanced Research laboratory, Hitachi, Ltd., 1995. <http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.160.5142>

[9] Rosenblum, Mendel; Ouserhout, John. ''The Design and Implementation of a Log-structured File System''. ''Association for Computing Machinery (ACM)''. University of California at Berkeley, Feb 1992. <http://portal.acm.org/citation.cfm?id=146943&coll=GUIDE&dl=GUIDE&CFID=108397378&CFTOKEN=72657973&ret=1#Fulltext>

[10] Shimpi, Anand. ''Intel X25-M SSD: Intel Delivers One of the World's Fastest Drives''. ''AnAndTech''. AnAndTech, 8 Sep 2008. <http://www.anandtech.com/show/2614>

[11] Shimpi, Anand. ''The SSD Relapse: Understanding and Choosing the Best SSD''. ''AnAndTech''. AnAndTech, 30 Aug 2009. <http://www.anandtech.com/show/2829>

[12] Shimpi, Anand. ''The SSD Anthology: Understanding SSDs and New Drives from OCZ''. ''AnAndTech''. AnAndTech, 18 Mar 2009. <http://www.anandtech.com/show/2738>

[13] Corbet, Jonathan. ''Solid-State Storage Devices and the Block Layer''. ''Linux Weekly News''. Linux Weekly News, 4 Oct 2010. <http://lwn.net/Articles/408428/>

[14] Woodhouse, David. ''JFFS : The Journalling Flash File System''. ''CiteSeerX''. Red Hat, Inc, Accessed 14 Oct 2010. <http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.128.6156&rep=rep1&type=pdf>

[15] Agrawal, Nitin; Prabhakaran, Vijayan; Wobber, Ted; Davis, John; Manasse, Mark. Panigrahy, Rina. ''Design Tradeoffs for SSD Performance''. ''Association for Computing Machinery (ACM)'', USENIX 2008 Annual Technical Conference, 2008. <http://portal.acm.org/citation.cfm?id=1404014.1404019>

[16] Lee, Sang-Won, et al. ''A Log Buffer-Based Flash Translation Layer Using Fully-Associative Sector Translation''. ''Association for Computing Machinery (ACM)''. ACM Transactions on Embedded Computing Systems (TECS), Jul 2007. <http://portal.acm.org/citation.cfm?id=1275990>

[17] ''Reach New Heights in Computing Performance''. ''Micron Technology Inc''. Micro Technology Inc, Accessed 14 Oct 2010. <http://www.micron.com/products/solid_state_storage/client_ssd.html>

[18] ''Flash Memories.'' 1 ed. New York: Springer, 1999. Print.

[19] ''Nonvolatile Memory Technologies with Emphasis on Flash: A Comprehensive Guide to Understanding and Using Flash Memory Devices''. ''IEEE Press Series on Microelectronic Systems''. New York: Wiley-Ieee Press, 2008. Print.

[20] ''Nonvolatile Semiconductor Memory Technology: A Comprehensive Guide to Understanding and Using NVSM Devices''. ''IEEE Press Series on Microelectronic Systems''. New York: Wiley-Ieee Press, 1997. Print.

=External links=

Relevant Wikipedia articles: [http://en.wikipedia.org/wiki/Flash_Memory Flash Memory], [http://en.wikipedia.org/wiki/LogFS LogFS], [http://en.wikipedia.org/wiki/Hard_disk Hard Disk Drives], [http://en.wikipedia.org/wiki/Wear_leveling Wear Leveling], [http://en.wikipedia.org/wiki/Hot_spot_%28computer_science%29 Hot Spots], [http://en.wikipedia.org/wiki/Solid-state_drive Sold State Drive].

COMP 3000 Essay 1 2010 Question 10

2010-10-15T05:10:18Z

Gsmith6: /* Answer */

=Question=

How do the constraints of flash storage affect the design of flash-optimized file systems? Explain by contrasting with hard disk-based file systems.

=Answer=
First introduced in the late 80s, Flash-memory is a light, energy-independent, compact, shock-resistant and efficiently readable type of storage. It started out as a replacement for EPROMs. At the time EPROMs needed a UV photoemission to be erased while flash memory could be erased electronically.[7] Because of the particular limitations of this kind of memory, flash file systems require a fundamentally different system architecture than disk-based file-systems: these systems need to be designed in light of flash-memory’s limited number of erase-cycles and its need to conduct erasures one entire block at a time. These constraints are a direct result of the same design that gives flash its advantages with regard to [ TO WHAT?] as both are due to [TO WHAT?] . Thus, a typical disk-based file-system is not suitable for working with flash memory as it erases far too frequently and indiscriminately while being simultaneously optimized for other constraints that do not affect flash memory. This means that a different solution is necessary and that solution is the log-based file-system which is far better suited to working with flash memory because it optimizes erasures by [WHAT?].

==Flash Memory==
Flash memory is non-volatile(meaning digital storage that does not require power to retain its memory) storage space that has become more popular recently due to its fast fetch times. There are two basic forms of the flash storage system, NOR and NAND. Each type has its advantages and disadvantages. NOR has the fastest read times, but is much slower at writing. NAND on the other hand has much more capacity, faster write times, is less expensive, and has a much longer life expectancy.[2]

'''Flash memory started out as a replacement for EPROMs. At the time EPROMs needed a UV photoemission to be erased while flash memory could be erased electronically. The first flash memory product came out in 1988 but it did not take off until the late 1990’s because it could not be reliable produced. NOR and NAND memory is named after the arrangement of the cells in the memory array. NOR based flash memory benefits from having very fast burst read times but slower write times. Due to the structure of NOR memory programs stored in NOR based memory can be executed without being loaded into RAM first. NAND flash memory has a very large storage capacity and can read and write large files relatively fast. NAND is more suited for storage while NOR memory is better suited for direct program execution such as in CMOS chips'''

More and more people use flash memory, with many sizes of drives, ranging from a few hundred megabyte USB key, to a few terabyte internal solid-state drive(SSD). Two main reasons for this movement are because of flash's extremely fast read times, and its falling price. A typical flash drive has read speeds of up to 14 times faster than a hard disk drive (HDD).[17]

This extreme read speed makes flash drives a preferred method of storing games. This effectively makes loading times virtually non-existent. There is however a downside to this method. Games constantly save, modify, and change files wearing out the blocks much quicker. Flash drives have been shown be effective for the use in web-servers for running their CSS scripts or HTML pages.

Although flash drives are exponentially faster than HDDs, they still have not become the main source of data management. The reason for this is because HDDs are simply much cheaper, and flash drives still have many faults. The most critical fault is that each block in flash memory can only be erased approximately 100,000 times.[14] This poses a problem because when modifying a file, even if its a single bit, the entire block must be erased, and rewritten. This erase/rewrite slows down the write operation considerably, making it actually slower to write a file to flash than an HDD.[8]

HDDs use a block system, in which the kernel specifies which blocks to read and write. When using a flash drive, the blocks are emulated and mapped to a physical memory address. It does through what is called a "Translational Layer".

==Traditionally Optimized File Systems==

Since the kernel asks for a block number, a conventional hard disk drive (HDD) file-system is not optimized to work with flash memory. The reason for this is that conventional hard-disks have different constraints from those of flash memory - their primary problem is to reduce seeking time, while the primary problem when working with flash memory is to erase in a minimal and balanced way.

The most consuming process for an HDD is seeking data by relocating the read-head and spinning the magnetic disk. A traditional file system optimizes the way it stores data by placing related blocks close-by on the disk in order to minimize mechanical movement within the HDD. One of the great advantages of flash memory, which accounts for its fast read speed, is that there is no need to seek data physically. This is also why defragmentaion, a procedure used by HDDs to put files into more convenient configurations and thus minimize seeking times, loses its purpose in a flash memory context. Indeed, the unnecessary erasures that it entails are both inefficient and harmful for a flash memory unit.

This comes directly out of flash memory's aforementioned constraints: the slow block-sized erasures and the limited number of erase-cycles. Because of these, a flash optimal file system needs to minimize its erase operations and also to spread out its erasures in such a way as to avoid the formation of hot-spots: sections of memory which have undergone a disproportionately high number of erasures and are thus in danger of burning out. This process of spreading out data is referred to as "wear leveling". To minimize hotspots, a system using flash memory would have to write new data to empty memory blocks. This method would also call for some sort of garbage collection to conduct necessary erasure operations while the system is idle. It makes better sense to do these at this time because of the slow nature of erasures in the flash memory context. Of course, there is no such feature in a traditional HDD file-system.

'''More on this later'''

==Flash Optimized File Systems==

The process of "wear leveling" ensures that he drive does not keep erasing and writing to the same block over and over. This is achieved by writing data that doesn’t change often to blocks that have been erased frequently. Wear leveling tries to make all the blocks use up there write cycles at an even pace increasing the overall life of the hard drive.[3] This is achieved through a Log-based File System, often referred to as the Flash Transitional Layer(FTL). Essentially, the drive stores a log that keeps track of how many times each erase sector has been invalidated (or erased). The translational layer has a translation table, where each physical memory address is associated with an emulated block sectors. This allows a traditional file system that uses block sectors to be used on the flash drive. Each block has a flag which keeps track of its state. When a block is being written to, the FTL marks the blocks needed as ''allocated''. This prevents other data being written to the block that has already been allocated. The FTL then goes on to write the data in the allocated blocks. Once it completes the transaction, the system updates the allocated blocks to ''pre-valid''. Once that is completed, the drive marks the invalidated blocks to ''invalid'', while marking the newly written block as ''valid''. This entire flagging process is to ensure that the newly allocated blocks are never mixed up with the invalidated blocks.

===Banks===
The FTL organizes data using structures called banks. When the FTL gets a request to write something to memory, it uses a bank list to determine which area of the drive should be used. Essentially a bank is a group of sequential addresses, that keeps track of when it was last updated using timestamps. The FTL will only write to that bank, and once there is not enough space to write anymore, it switches out the current bank for the one with the most available space. When cleaning up the bank, the system puts it into what is called the Cleaning Bank List and removes it from the Bank List, thus avoiding any chance of some data being written to that bank while something is being erased.

===Cleaner===
When the FTL realizes that there is not enough room to write new data onto the drive, it runs a garbage collection routine. This routine selects a segment to be cleaned, copies all of the valid data into a new segment, then erases everything in the old segment. This frees up the otherwise useless invalidated blocks and by not erasing every block as soon as it becomes invalidated, it saves on the amount of times that the expensive erase operation is called. The kernel can preemptively clean the drive when the system is idle.

==Conclusion==

In this way, thanks to its [WHATEVER MAKES LOG FSs ACTUALLY GOOD AT DEALING WITH FLASH], the log-based file-system is far better suited to working with flash memory than a traditional HDD file system. The latter is utterly unfit for this task due to its placing primacy on the minimization of seeks rather than on the minimization and management of erasures. Dealing smartly with erasures is extremely important for a flash memory file system, as that memory type's particular weaknesses, the limited number of erasure cycles, the necessity to erase by the block and the relative slowness of the erasures themselves, all have to do with erasing. A good flash memory file system must therefore be built with the aim of making the best of these weaknesses and this is precisely the reason why older disk-based file systems are not suitable for flash memory while log-based file systems are. [INSPIRATIONAL LAST WORDS]

=Questions=

=References=

[1] Kim, Han-joon; Lee, Sang-goo. ''A New Flash Memory Management for Flash Storage System''. ''IEEExplore''. Dept. of Comput. Sci., Seoul Nat. Univ., 06 Aug 2002. <http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=812717&tag=1#>

[2] Smith, Lance. ''NAND Flash Solid State Storage Performance and Capability''. ''Flash Memory Summit''. SNIA Education Committee, 18 Aug 2009. <http://www.flashmemorysummit.com/English/Collaterals/Proceedings/2009/20090812_T1B_Smith.pdf>

[3] Chang, LiPin. ''On Efficient Wear Leveling for Large-Scale Flash-Memory Storage Systems''. ''Association for Computing Machinery (ACM)''. Dept. of Comput. Sci.,Nat. ChiaoTung Univ., 15 Mar 2007. <http://portal.acm.org/citation.cfm?id=1244248>

[4] Nath, Suman; Gibbons, Phillip. ''Online maintenance of very large random samples on flash storage''. ''Association for Computing Machinery (ACM)''. The VLDB Journal, 27 Jul 2007. <http://portal.acm.org/citation.cfm?id=1731355>

[5] Lim, Seung-Ho; Park; Kyu-Ho. ''An Efficient NAND Flash File System for Flash Memory Storage''. ''CORE Laboratory''. IEEE Transactions On Computers, Jul 2006. <http://vlsi.kaist.ac.kr/paper_list/2006_TC_CFFS.pdf>

[6] ''NAND vs. NOR Flash Memory Technology Overview''. ''RMG and Associates''. Toshiba America, accessed 14 Oct 2010. <http://maltiel-consulting.com/NAND_vs_NOR_Flash_Memory_Technology_Overview_Read_Write_Erase_speed_for_SLC_MLC_semiconductor_consulting_expert.pdf>

[7] Bez, Roberto; Camerlenghi, Emilio; Modelli, Alberto; Visconti, Angelo. ''Introduction to Flash Memory''. ''IEEExplore''. STMicroelectronics, 21 May 2003. <http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1199079&tag=1>

[8] Kawaguchi, Atsuo; Nishioka, Shingo; Motoda Hiroshi. ''A Flash-Memory Based File System''. ''CiteSeerX'' Advanced Research laboratory, Hitachi, Ltd., 1995. <http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.160.5142>

[9] Rosenblum, Mendel; Ouserhout, John. ''The Design and Implementation of a Log-structured File System''. ''Association for Computing Machinery (ACM)''. University of California at Berkeley, Feb 1992. <http://portal.acm.org/citation.cfm?id=146943&coll=GUIDE&dl=GUIDE&CFID=108397378&CFTOKEN=72657973&ret=1#Fulltext>

[10] Shimpi, Anand. ''Intel X25-M SSD: Intel Delivers One of the World's Fastest Drives''. ''AnAndTech''. AnAndTech, 8 Sep 2008. <http://www.anandtech.com/show/2614>

[11] Shimpi, Anand. ''The SSD Relapse: Understanding and Choosing the Best SSD''. ''AnAndTech''. AnAndTech, 30 Aug 2009. <http://www.anandtech.com/show/2829>

[12] Shimpi, Anand. ''The SSD Anthology: Understanding SSDs and New Drives from OCZ''. ''AnAndTech''. AnAndTech, 18 Mar 2009. <http://www.anandtech.com/show/2738>

[13] Corbet, Jonathan. ''Solid-State Storage Devices and the Block Layer''. ''Linux Weekly News''. Linux Weekly News, 4 Oct 2010. <http://lwn.net/Articles/408428/>

[14] Woodhouse, David. ''JFFS : The Journalling Flash File System''. ''CiteSeerX''. Red Hat, Inc, Accessed 14 Oct 2010. <http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.128.6156&rep=rep1&type=pdf>

[15] Agrawal, Nitin; Prabhakaran, Vijayan; Wobber, Ted; Davis, John; Manasse, Mark. Panigrahy, Rina. ''Design Tradeoffs for SSD Performance''. ''Association for Computing Machinery (ACM)'', USENIX 2008 Annual Technical Conference, 2008. <http://portal.acm.org/citation.cfm?id=1404014.1404019>

[16] Lee, Sang-Won, et al. ''A Log Buffer-Based Flash Translation Layer Using Fully-Associative Sector Translation''. ''Association for Computing Machinery (ACM)''. ACM Transactions on Embedded Computing Systems (TECS), Jul 2007. <http://portal.acm.org/citation.cfm?id=1275990>

[17] ''Reach New Heights in Computing Performance''. ''Micron Technology Inc''. Micro Technology Inc, Accessed 14 Oct 2010. <http://www.micron.com/products/solid_state_storage/client_ssd.html>

[18] ''Flash Memories.'' 1 ed. New York: Springer, 1999. Print.

[19] ''Nonvolatile Memory Technologies with Emphasis on Flash: A Comprehensive Guide to Understanding and Using Flash Memory Devices''. ''IEEE Press Series on Microelectronic Systems''. New York: Wiley-Ieee Press, 2008. Print.

[20] ''Nonvolatile Semiconductor Memory Technology: A Comprehensive Guide to Understanding and Using NVSM Devices''. ''IEEE Press Series on Microelectronic Systems''. New York: Wiley-Ieee Press, 1997. Print.

=External links=

Relevant Wikipedia articles: [http://en.wikipedia.org/wiki/Flash_Memory Flash Memory], [http://en.wikipedia.org/wiki/LogFS LogFS], [http://en.wikipedia.org/wiki/Hard_disk Hard Disk Drives], [http://en.wikipedia.org/wiki/Wear_leveling Wear Leveling], [http://en.wikipedia.org/wiki/Hot_spot_%28computer_science%29 Hot Spots], [http://en.wikipedia.org/wiki/Solid-state_drive Sold State Drive].

COMP 3000 Essay 1 2010 Question 10

2010-10-15T05:02:50Z

Gsmith6: /* Conclusion */

=Question=

How do the constraints of flash storage affect the design of flash-optimized file systems? Explain by contrasting with hard disk-based file systems.

=Answer=
First introduced in the late 80s, Flash-memory is a light, energy-independent, compact, shock-resistant and efficiently readable type of storage. It started out as a replacement for EPROMs. At the time EPROMs needed a UV photoemission to be erased while flash memory could be erased electronically.[7] Because of the particular limitations of this kind of memory, flash file systems require a fundamentally different system architecture than disk-based file-systems: these systems need to be designed in light of flash-memory’s limited number of erase-cycles and its need to conduct erasures one entire block at a time. These constraints are a direct result of the same design that gives flash its advantages with regard to [ TO WHAT?] as both are due to [TO WHAT?] . Thus, a typical disk-based file-system is not suitable for working with flash memory as it erases far too frequently and indiscriminately while being simultaneously optimized for other constraints that do not affect flash memory. This means that a different solution is necessary and that solution is the log-based file-system which is far better suited to working with flash memory because it optimizes erasures by [WHAT?].

==Flash Memory==
Flash memory is non-volatile(meaning digital storage that does not require power to retain its memory) storage space that has become more popular recently due to its fast fetch times. There are two basic forms of the flash storage system, NOR and NAND. Each type has its advantages and disadvantages. NOR has the fastest read times, but is much slower at writing. NAND on the other hand has much more capacity, faster write times, is less expensive, and has a much longer life expectancy.[2]

More and more people use flash memory, with many sizes of drives, ranging from a few hundred megabyte USB key, to a few terabyte internal solid-state drive(SSD). Two main reasons for this movement are because of flash's extremely fast read times, and its falling price. A typical flash drive has read speeds of up to 14 times faster than a hard disk drive (HDD).[17]

Although flash drives are exponentially faster than HDDs, they still have not become the main source of data management. The reason for this is because HDDs are simply much cheaper, and flash drives still have many faults. The most critical fault is that each block in flash memory can only be erased approximately 100,000 times.[14] This poses a problem because when modifying a file, even if its a single bit, the entire block must be erased, and rewritten. This erase/rewrite slows down the write operation considerably, making it actually slower to write a file to flash than an HDD.[8]

This extreme read speed makes flash drives a preferred method of storing games. This effectively makes loading times virtually non-existent. There is however a downside to this method. Games constantly save, modify, and change files wearing out the blocks much quicker. Flash drives have been shown be effective for the use in web-servers for running their CSS scripts or HTML pages.

'''Flash memory started out as a replacement for EPROMs. At the time EPROMs needed a UV photoemission to be erased while flash memory could be erased electronically. The first flash memory product came out in 1988 but it did not take off until the late 1990’s because it could not be reliable produced. NOR and NAND memory is named after the arrangement of the cells in the memory array. NOR based flash memory benefits from having very fast burst read times but slower write times. Due to the structure of NOR memory programs stored in NOR based memory can be executed without being loaded into RAM first. NAND flash memory has a very large storage capacity and can read and write large files relatively fast. NAND is more suited for storage while NOR memory is better suited for direct program execution such as in CMOS chips'''

HDDs use a block system, in which the kernel specifies which blocks to read and write. When using a flash drive, the blocks are emulated and mapped to a physical memory address. It does through what is called a "Translational Layer".

==Traditionally Optimized File Systems==

Since the kernel asks for a block number, a conventional hard disk drive (HDD) file-system is not optimized to work with flash memory. The reason for this is that conventional hard-disks have different constraints from those of flash memory - their primary problem is to reduce seeking time, while the primary problem when working with flash memory is to erase in a minimal and balanced way.

The most consuming process for an HDD is seeking data by relocating the read-head and spinning the magnetic disk. A traditional file system optimizes the way it stores data by placing related blocks close-by on the disk in order to minimize mechanical movement within the HDD. One of the great advantages of flash memory, which accounts for its fast read speed, is that there is no need to seek data physically. This is also why defragmentaion, a procedure used by HDDs to put files into more convenient configurations and thus minimize seeking times, loses its purpose in a flash memory context. Indeed, the unnecessary erasures that it entails are both inefficient and harmful for a flash memory unit.

This comes directly out of flash memory's aforementioned constraints: the slow block-sized erasures and the limited number of erase-cycles. Because of these, a flash optimal file system needs to minimize its erase operations and also to spread out its erasures in such a way as to avoid the formation of hot-spots: sections of memory which have undergone a disproportionately high number of erasures and are thus in danger of burning out. This process of spreading out data is referred to as "wear leveling". To minimize hotspots, a system using flash memory would have to write new data to empty memory blocks. This method would also call for some sort of garbage collection to conduct necessary erasure operations while the system is idle. It makes better sense to do these at this time because of the slow nature of erasures in the flash memory context. Of course, there is no such feature in a traditional HDD file-system.

'''More on this later'''

==Flash Optimized File Systems==

The process of "wear leveling" ensures that he drive does not keep erasing and writing to the same block over and over. This is achieved by writing data that doesn’t change often to blocks that have been erased frequently. Wear leveling tries to make all the blocks use up there write cycles at an even pace increasing the overall life of the hard drive.[3] This is achieved through a Log-based File System, often referred to as the Flash Transitional Layer(FTL). Essentially, the drive stores a log that keeps track of how many times each erase sector has been invalidated (or erased). The translational layer has a translation table, where each physical memory address is associated with an emulated block sectors. This allows a traditional file system that uses block sectors to be used on the flash drive. Each block has a flag which keeps track of its state. When a block is being written to, the FTL marks the blocks needed as ''allocated''. This prevents other data being written to the block that has already been allocated. The FTL then goes on to write the data in the allocated blocks. Once it completes the transaction, the system updates the allocated blocks to ''pre-valid''. Once that is completed, the drive marks the invalidated blocks to ''invalid'', while marking the newly written block as ''valid''. This entire flagging process is to ensure that the newly allocated blocks are never mixed up with the invalidated blocks.

===Banks===
The FTL organizes data using structures called banks. When the FTL gets a request to write something to memory, it uses a bank list to determine which area of the drive should be used. Essentially a bank is a group of sequential addresses, that keeps track of when it was last updated using timestamps. The FTL will only write to that bank, and once there is not enough space to write anymore, it switches out the current bank for the one with the most available space. When cleaning up the bank, the system puts it into what is called the Cleaning Bank List and removes it from the Bank List, thus avoiding any chance of some data being written to that bank while something is being erased.

===Cleaner===
When the FTL realizes that there is not enough room to write new data onto the drive, it runs a garbage collection routine. This routine selects a segment to be cleaned, copies all of the valid data into a new segment, then erases everything in the old segment. This frees up the otherwise useless invalidated blocks and by not erasing every block as soon as it becomes invalidated, it saves on the amount of times that the expensive erase operation is called. The kernel can preemptively clean the drive when the system is idle.

==Conclusion==

In this way, thanks to its [WHATEVER MAKES LOG FSs ACTUALLY GOOD AT DEALING WITH FLASH], the log-based file-system is far better suited to working with flash memory than a traditional HDD file system. The latter is utterly unfit for this task due to its placing primacy on the minimization of seeks rather than on the minimization and management of erasures. Dealing smartly with erasures is extremely important for a flash memory file system, as that memory type's particular weaknesses, the limited number of erasure cycles, the necessity to erase by the block and the relative slowness of the erasures themselves, all have to do with erasing. A good flash memory file system must therefore be built with the aim of making the best of these weaknesses and this is precisely the reason why older disk-based file systems are not suitable for flash memory while log-based file systems are. [INSPIRATIONAL LAST WORDS]

=Questions=

=References=

[1] Kim, Han-joon; Lee, Sang-goo. ''A New Flash Memory Management for Flash Storage System''. ''IEEExplore''. Dept. of Comput. Sci., Seoul Nat. Univ., 06 Aug 2002. <http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=812717&tag=1#>

[2] Smith, Lance. ''NAND Flash Solid State Storage Performance and Capability''. ''Flash Memory Summit''. SNIA Education Committee, 18 Aug 2009. <http://www.flashmemorysummit.com/English/Collaterals/Proceedings/2009/20090812_T1B_Smith.pdf>

[3] Chang, LiPin. ''On Efficient Wear Leveling for Large-Scale Flash-Memory Storage Systems''. ''Association for Computing Machinery (ACM)''. Dept. of Comput. Sci.,Nat. ChiaoTung Univ., 15 Mar 2007. <http://portal.acm.org/citation.cfm?id=1244248>

[4] Nath, Suman; Gibbons, Phillip. ''Online maintenance of very large random samples on flash storage''. ''Association for Computing Machinery (ACM)''. The VLDB Journal, 27 Jul 2007. <http://portal.acm.org/citation.cfm?id=1731355>

[5] Lim, Seung-Ho; Park; Kyu-Ho. ''An Efficient NAND Flash File System for Flash Memory Storage''. ''CORE Laboratory''. IEEE Transactions On Computers, Jul 2006. <http://vlsi.kaist.ac.kr/paper_list/2006_TC_CFFS.pdf>

[6] ''NAND vs. NOR Flash Memory Technology Overview''. ''RMG and Associates''. Toshiba America, accessed 14 Oct 2010. <http://maltiel-consulting.com/NAND_vs_NOR_Flash_Memory_Technology_Overview_Read_Write_Erase_speed_for_SLC_MLC_semiconductor_consulting_expert.pdf>

[7] Bez, Roberto; Camerlenghi, Emilio; Modelli, Alberto; Visconti, Angelo. ''Introduction to Flash Memory''. ''IEEExplore''. STMicroelectronics, 21 May 2003. <http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1199079&tag=1>

[8] Kawaguchi, Atsuo; Nishioka, Shingo; Motoda Hiroshi. ''A Flash-Memory Based File System''. ''CiteSeerX'' Advanced Research laboratory, Hitachi, Ltd., 1995. <http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.160.5142>

[9] Rosenblum, Mendel; Ouserhout, John. ''The Design and Implementation of a Log-structured File System''. ''Association for Computing Machinery (ACM)''. University of California at Berkeley, Feb 1992. <http://portal.acm.org/citation.cfm?id=146943&coll=GUIDE&dl=GUIDE&CFID=108397378&CFTOKEN=72657973&ret=1#Fulltext>

[10] Shimpi, Anand. ''Intel X25-M SSD: Intel Delivers One of the World's Fastest Drives''. ''AnAndTech''. AnAndTech, 8 Sep 2008. <http://www.anandtech.com/show/2614>

[11] Shimpi, Anand. ''The SSD Relapse: Understanding and Choosing the Best SSD''. ''AnAndTech''. AnAndTech, 30 Aug 2009. <http://www.anandtech.com/show/2829>

[12] Shimpi, Anand. ''The SSD Anthology: Understanding SSDs and New Drives from OCZ''. ''AnAndTech''. AnAndTech, 18 Mar 2009. <http://www.anandtech.com/show/2738>

[13] Corbet, Jonathan. ''Solid-State Storage Devices and the Block Layer''. ''Linux Weekly News''. Linux Weekly News, 4 Oct 2010. <http://lwn.net/Articles/408428/>

[14] Woodhouse, David. ''JFFS : The Journalling Flash File System''. ''CiteSeerX''. Red Hat, Inc, Accessed 14 Oct 2010. <http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.128.6156&rep=rep1&type=pdf>

[15] Agrawal, Nitin; Prabhakaran, Vijayan; Wobber, Ted; Davis, John; Manasse, Mark. Panigrahy, Rina. ''Design Tradeoffs for SSD Performance''. ''Association for Computing Machinery (ACM)'', USENIX 2008 Annual Technical Conference, 2008. <http://portal.acm.org/citation.cfm?id=1404014.1404019>

[16] Lee, Sang-Won, et al. ''A Log Buffer-Based Flash Translation Layer Using Fully-Associative Sector Translation''. ''Association for Computing Machinery (ACM)''. ACM Transactions on Embedded Computing Systems (TECS), Jul 2007. <http://portal.acm.org/citation.cfm?id=1275990>

[17] ''Reach New Heights in Computing Performance''. ''Micron Technology Inc''. Micro Technology Inc, Accessed 14 Oct 2010. <http://www.micron.com/products/solid_state_storage/client_ssd.html>

[18] ''Flash Memories.'' 1 ed. New York: Springer, 1999. Print.

[19] ''Nonvolatile Memory Technologies with Emphasis on Flash: A Comprehensive Guide to Understanding and Using Flash Memory Devices''. ''IEEE Press Series on Microelectronic Systems''. New York: Wiley-Ieee Press, 2008. Print.

[20] ''Nonvolatile Semiconductor Memory Technology: A Comprehensive Guide to Understanding and Using NVSM Devices''. ''IEEE Press Series on Microelectronic Systems''. New York: Wiley-Ieee Press, 1997. Print.

=External links=

Relevant Wikipedia articles: [http://en.wikipedia.org/wiki/Flash_Memory Flash Memory], [http://en.wikipedia.org/wiki/LogFS LogFS], [http://en.wikipedia.org/wiki/Hard_disk Hard Disk Drives], [http://en.wikipedia.org/wiki/Wear_leveling Wear Leveling], [http://en.wikipedia.org/wiki/Hot_spot_%28computer_science%29 Hot Spots], [http://en.wikipedia.org/wiki/Solid-state_drive Sold State Drive].

COMP 3000 Essay 1 2010 Question 10

2010-10-15T05:02:11Z

Gsmith6: /* Flash Memory */

=Question=

How do the constraints of flash storage affect the design of flash-optimized file systems? Explain by contrasting with hard disk-based file systems.

=Answer=
First introduced in the late 80s, Flash-memory is a light, energy-independent, compact, shock-resistant and efficiently readable type of storage. It started out as a replacement for EPROMs. At the time EPROMs needed a UV photoemission to be erased while flash memory could be erased electronically.[7] Because of the particular limitations of this kind of memory, flash file systems require a fundamentally different system architecture than disk-based file-systems: these systems need to be designed in light of flash-memory’s limited number of erase-cycles and its need to conduct erasures one entire block at a time. These constraints are a direct result of the same design that gives flash its advantages with regard to [ TO WHAT?] as both are due to [TO WHAT?] . Thus, a typical disk-based file-system is not suitable for working with flash memory as it erases far too frequently and indiscriminately while being simultaneously optimized for other constraints that do not affect flash memory. This means that a different solution is necessary and that solution is the log-based file-system which is far better suited to working with flash memory because it optimizes erasures by [WHAT?].

==Flash Memory==
Flash memory is non-volatile(meaning digital storage that does not require power to retain its memory) storage space that has become more popular recently due to its fast fetch times. There are two basic forms of the flash storage system, NOR and NAND. Each type has its advantages and disadvantages. NOR has the fastest read times, but is much slower at writing. NAND on the other hand has much more capacity, faster write times, is less expensive, and has a much longer life expectancy.[2]

More and more people use flash memory, with many sizes of drives, ranging from a few hundred megabyte USB key, to a few terabyte internal solid-state drive(SSD). Two main reasons for this movement are because of flash's extremely fast read times, and its falling price. A typical flash drive has read speeds of up to 14 times faster than a hard disk drive (HDD).[17]

Although flash drives are exponentially faster than HDDs, they still have not become the main source of data management. The reason for this is because HDDs are simply much cheaper, and flash drives still have many faults. The most critical fault is that each block in flash memory can only be erased approximately 100,000 times.[14] This poses a problem because when modifying a file, even if its a single bit, the entire block must be erased, and rewritten. This erase/rewrite slows down the write operation considerably, making it actually slower to write a file to flash than an HDD.[8]

This extreme read speed makes flash drives a preferred method of storing games. This effectively makes loading times virtually non-existent. There is however a downside to this method. Games constantly save, modify, and change files wearing out the blocks much quicker. Flash drives have been shown be effective for the use in web-servers for running their CSS scripts or HTML pages.

'''Flash memory started out as a replacement for EPROMs. At the time EPROMs needed a UV photoemission to be erased while flash memory could be erased electronically. The first flash memory product came out in 1988 but it did not take off until the late 1990’s because it could not be reliable produced. NOR and NAND memory is named after the arrangement of the cells in the memory array. NOR based flash memory benefits from having very fast burst read times but slower write times. Due to the structure of NOR memory programs stored in NOR based memory can be executed without being loaded into RAM first. NAND flash memory has a very large storage capacity and can read and write large files relatively fast. NAND is more suited for storage while NOR memory is better suited for direct program execution such as in CMOS chips'''

HDDs use a block system, in which the kernel specifies which blocks to read and write. When using a flash drive, the blocks are emulated and mapped to a physical memory address. It does through what is called a "Translational Layer".

==Traditionally Optimized File Systems==

Since the kernel asks for a block number, a conventional hard disk drive (HDD) file-system is not optimized to work with flash memory. The reason for this is that conventional hard-disks have different constraints from those of flash memory - their primary problem is to reduce seeking time, while the primary problem when working with flash memory is to erase in a minimal and balanced way.

The most consuming process for an HDD is seeking data by relocating the read-head and spinning the magnetic disk. A traditional file system optimizes the way it stores data by placing related blocks close-by on the disk in order to minimize mechanical movement within the HDD. One of the great advantages of flash memory, which accounts for its fast read speed, is that there is no need to seek data physically. This is also why defragmentaion, a procedure used by HDDs to put files into more convenient configurations and thus minimize seeking times, loses its purpose in a flash memory context. Indeed, the unnecessary erasures that it entails are both inefficient and harmful for a flash memory unit.

This comes directly out of flash memory's aforementioned constraints: the slow block-sized erasures and the limited number of erase-cycles. Because of these, a flash optimal file system needs to minimize its erase operations and also to spread out its erasures in such a way as to avoid the formation of hot-spots: sections of memory which have undergone a disproportionately high number of erasures and are thus in danger of burning out. This process of spreading out data is referred to as "wear leveling". To minimize hotspots, a system using flash memory would have to write new data to empty memory blocks. This method would also call for some sort of garbage collection to conduct necessary erasure operations while the system is idle. It makes better sense to do these at this time because of the slow nature of erasures in the flash memory context. Of course, there is no such feature in a traditional HDD file-system.

'''More on this later'''

==Flash Optimized File Systems==

The process of "wear leveling" ensures that he drive does not keep erasing and writing to the same block over and over. This is achieved by writing data that doesn’t change often to blocks that have been erased frequently. Wear leveling tries to make all the blocks use up there write cycles at an even pace increasing the overall life of the hard drive.[3] This is achieved through a Log-based File System, often referred to as the Flash Transitional Layer(FTL). Essentially, the drive stores a log that keeps track of how many times each erase sector has been invalidated (or erased). The translational layer has a translation table, where each physical memory address is associated with an emulated block sectors. This allows a traditional file system that uses block sectors to be used on the flash drive. Each block has a flag which keeps track of its state. When a block is being written to, the FTL marks the blocks needed as ''allocated''. This prevents other data being written to the block that has already been allocated. The FTL then goes on to write the data in the allocated blocks. Once it completes the transaction, the system updates the allocated blocks to ''pre-valid''. Once that is completed, the drive marks the invalidated blocks to ''invalid'', while marking the newly written block as ''valid''. This entire flagging process is to ensure that the newly allocated blocks are never mixed up with the invalidated blocks.

===Banks===
The FTL organizes data using structures called banks. When the FTL gets a request to write something to memory, it uses a bank list to determine which area of the drive should be used. Essentially a bank is a group of sequential addresses, that keeps track of when it was last updated using timestamps. The FTL will only write to that bank, and once there is not enough space to write anymore, it switches out the current bank for the one with the most available space. When cleaning up the bank, the system puts it into what is called the Cleaning Bank List and removes it from the Bank List, thus avoiding any chance of some data being written to that bank while something is being erased.

===Cleaner===
When the FTL realizes that there is not enough room to write new data onto the drive, it runs a garbage collection routine. This routine selects a segment to be cleaned, copies all of the valid data into a new segment, then erases everything in the old segment. This frees up the otherwise useless invalidated blocks and by not erasing every block as soon as it becomes invalidated, it saves on the amount of times that the expensive erase operation is called. The kernel can preemptively clean the drive when the system is idle.

=Conclusion=

In this way, thanks to its [WHATEVER MAKES LOG FSs ACTUALLY GOOD AT DEALING WITH FLASH], the log-based file-system is far better suited to working with flash memory than a traditional HDD file system. The latter is utterly unfit for this task due to its placing primacy on the minimization of seeks rather than on the minimization and management of erasures. Dealing smartly with erasures is extremely important for a flash memory file system, as that memory type's particular weaknesses, the limited number of erasure cycles, the necessity to erase by the block and the relative slowness of the erasures themselves, all have to do with erasing. A good flash memory file system must therefore be built with the aim of making the best of these weaknesses and this is precisely the reason why older disk-based file systems are not suitable for flash memory while log-based file systems are. [INSPIRATIONAL LAST WORDS]

=Questions=

=References=

[1] Kim, Han-joon; Lee, Sang-goo. ''A New Flash Memory Management for Flash Storage System''. ''IEEExplore''. Dept. of Comput. Sci., Seoul Nat. Univ., 06 Aug 2002. <http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=812717&tag=1#>

[2] Smith, Lance. ''NAND Flash Solid State Storage Performance and Capability''. ''Flash Memory Summit''. SNIA Education Committee, 18 Aug 2009. <http://www.flashmemorysummit.com/English/Collaterals/Proceedings/2009/20090812_T1B_Smith.pdf>

[3] Chang, LiPin. ''On Efficient Wear Leveling for Large-Scale Flash-Memory Storage Systems''. ''Association for Computing Machinery (ACM)''. Dept. of Comput. Sci.,Nat. ChiaoTung Univ., 15 Mar 2007. <http://portal.acm.org/citation.cfm?id=1244248>

[4] Nath, Suman; Gibbons, Phillip. ''Online maintenance of very large random samples on flash storage''. ''Association for Computing Machinery (ACM)''. The VLDB Journal, 27 Jul 2007. <http://portal.acm.org/citation.cfm?id=1731355>

[5] Lim, Seung-Ho; Park; Kyu-Ho. ''An Efficient NAND Flash File System for Flash Memory Storage''. ''CORE Laboratory''. IEEE Transactions On Computers, Jul 2006. <http://vlsi.kaist.ac.kr/paper_list/2006_TC_CFFS.pdf>

[6] ''NAND vs. NOR Flash Memory Technology Overview''. ''RMG and Associates''. Toshiba America, accessed 14 Oct 2010. <http://maltiel-consulting.com/NAND_vs_NOR_Flash_Memory_Technology_Overview_Read_Write_Erase_speed_for_SLC_MLC_semiconductor_consulting_expert.pdf>

[7] Bez, Roberto; Camerlenghi, Emilio; Modelli, Alberto; Visconti, Angelo. ''Introduction to Flash Memory''. ''IEEExplore''. STMicroelectronics, 21 May 2003. <http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1199079&tag=1>

[8] Kawaguchi, Atsuo; Nishioka, Shingo; Motoda Hiroshi. ''A Flash-Memory Based File System''. ''CiteSeerX'' Advanced Research laboratory, Hitachi, Ltd., 1995. <http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.160.5142>

[9] Rosenblum, Mendel; Ouserhout, John. ''The Design and Implementation of a Log-structured File System''. ''Association for Computing Machinery (ACM)''. University of California at Berkeley, Feb 1992. <http://portal.acm.org/citation.cfm?id=146943&coll=GUIDE&dl=GUIDE&CFID=108397378&CFTOKEN=72657973&ret=1#Fulltext>

[10] Shimpi, Anand. ''Intel X25-M SSD: Intel Delivers One of the World's Fastest Drives''. ''AnAndTech''. AnAndTech, 8 Sep 2008. <http://www.anandtech.com/show/2614>

[11] Shimpi, Anand. ''The SSD Relapse: Understanding and Choosing the Best SSD''. ''AnAndTech''. AnAndTech, 30 Aug 2009. <http://www.anandtech.com/show/2829>

[12] Shimpi, Anand. ''The SSD Anthology: Understanding SSDs and New Drives from OCZ''. ''AnAndTech''. AnAndTech, 18 Mar 2009. <http://www.anandtech.com/show/2738>

[13] Corbet, Jonathan. ''Solid-State Storage Devices and the Block Layer''. ''Linux Weekly News''. Linux Weekly News, 4 Oct 2010. <http://lwn.net/Articles/408428/>

[14] Woodhouse, David. ''JFFS : The Journalling Flash File System''. ''CiteSeerX''. Red Hat, Inc, Accessed 14 Oct 2010. <http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.128.6156&rep=rep1&type=pdf>

[15] Agrawal, Nitin; Prabhakaran, Vijayan; Wobber, Ted; Davis, John; Manasse, Mark. Panigrahy, Rina. ''Design Tradeoffs for SSD Performance''. ''Association for Computing Machinery (ACM)'', USENIX 2008 Annual Technical Conference, 2008. <http://portal.acm.org/citation.cfm?id=1404014.1404019>

[16] Lee, Sang-Won, et al. ''A Log Buffer-Based Flash Translation Layer Using Fully-Associative Sector Translation''. ''Association for Computing Machinery (ACM)''. ACM Transactions on Embedded Computing Systems (TECS), Jul 2007. <http://portal.acm.org/citation.cfm?id=1275990>

[17] ''Reach New Heights in Computing Performance''. ''Micron Technology Inc''. Micro Technology Inc, Accessed 14 Oct 2010. <http://www.micron.com/products/solid_state_storage/client_ssd.html>

[18] ''Flash Memories.'' 1 ed. New York: Springer, 1999. Print.

[19] ''Nonvolatile Memory Technologies with Emphasis on Flash: A Comprehensive Guide to Understanding and Using Flash Memory Devices''. ''IEEE Press Series on Microelectronic Systems''. New York: Wiley-Ieee Press, 2008. Print.

[20] ''Nonvolatile Semiconductor Memory Technology: A Comprehensive Guide to Understanding and Using NVSM Devices''. ''IEEE Press Series on Microelectronic Systems''. New York: Wiley-Ieee Press, 1997. Print.

=External links=

Relevant Wikipedia articles: [http://en.wikipedia.org/wiki/Flash_Memory Flash Memory], [http://en.wikipedia.org/wiki/LogFS LogFS], [http://en.wikipedia.org/wiki/Hard_disk Hard Disk Drives], [http://en.wikipedia.org/wiki/Wear_leveling Wear Leveling], [http://en.wikipedia.org/wiki/Hot_spot_%28computer_science%29 Hot Spots], [http://en.wikipedia.org/wiki/Solid-state_drive Sold State Drive].

COMP 3000 Essay 1 2010 Question 10

2010-10-15T03:26:12Z

Gsmith6: /* Flash Memory */

=Question=

How do the constraints of flash storage affect the design of flash-optimized file systems? Explain by contrasting with hard disk-based file systems.

=Answer=
First introduced in the late 80s, Flash-memory is a light, energy-independent, compact, shock-resistant and efficiently readable type of storage. Because of the particular limitations of this kind of memory, flash file systems require a fundamentally different system architecture than disk-based file-systems: these systems need to be designed in light of flash-memory’s limited number of erase-cycles and its need to conduct erasures one entire block at a time. These constraints are a direct result of the same design that gives flash its advantages with regard to [ TO WHAT?] as both are due to [TO WHAT?] . Thus, a typical disk-based file-system is not suitable for working with flash memory as it erases far too frequently and indiscriminately while being simultaneously optimized for other constraints that do not affect flash memory. This means that a different solution is necessary and that solution is the log-based file-system which is far better suited to working with flash memory because it optimizes erasures by [WHAT?].

==Flash Memory==
Flash memory is non-volatile(meaning digital storage that does not require power to retain its memory) storage space that has become more popular recently due to its fast fetch times. There are two basic forms of the flash storage system, NOR and NAND. Each type has its advantages and disadvantages. NOR has the fastest read times, but is much slower at writing. NAND on the other hand has much more capacity, faster write times, is less expensive, and has a much longer life expectancy.[2]

More and more people use flash memory, with many sizes of drives, ranging from a few hundred megabyte USB key, to a few terabyte internal solid-state drive(SSD). Two main reasons for this movement are because of flash's extremely fast read times, and its falling price. A typical flash drive has read speeds of up to 14 times faster than a hard disk drive (HDD).[17]

Although flash drives are exponentially faster than HDDs, they still have not become the main source of data management. The reason for this is because HDDs are simply much cheaper, and flash drives still have many faults. The most critical fault is that each block in flash memory can only be erased approximately 100,000 times.[14] This poses a problem because when modifying a file, even if its a single bit, the entire block must be erased, and rewritten. This erase/rewrite slows down the write operation considerably, making it actually slower to write a file to flash than an HDD.[8]

'''1. What flash storage is, why its good but also why it must have the problems that it does (the assumption is that it must have them, why would it otherwise?) [don't know much about this just now... basics include that there is NOR (reads slightly faster)and NAND (holds more, writes faster, erases much faster, lasts about ten times longer) flash with NAND being especially popular for storage (what's NOR good for?). Here, we'd ideally want to talk about why flash was invented (supposed as an alternative to slow ROM), why it was suitable for that, and how it works on a technical level. Then, we'd want to mention why this technical functionality was innovative and useful but also why it came with two serious set-backs: having a limited-number of re-write cycles and needing to erase a block at a time.'''
'''Either way, Flash storage affords far faster fetch times than the traditional platter-based HDD, and stability of information in a sense. Where the data is not actually stored, but reprogrammed, in a sense, the data is more secure and is less likely to be erased easily. On that note, in order to flip a single bit, that entire block will need to be erased, then reprogrammed. In an 'old' HDD, let's say, When the HDD fails at the end of its life cycle, your data is gone. (unless you're willing to shell out $200/hr to have it recovered, yes I've seen companies in Ottawa that do this.) In a flash HDD, when it reaches the end of its life, it merely becomes read-only. Bugger for Databases, but useful for technical notes and archives, let's say. With today's modern gaming computers, Flash memory can be good on quick load times, however with limited read-writes, it could afford better use to things that are not updated as frequently. I.e... Well I don't have a better example than a webserver hosting a company's CSS and scripts. ~Source: Years in the 'biz'''
'''Flash memory started out as a replacement for EPROMs. At the time EPROMs needed a UV photoemission to be erased while flash memory could be erased electronically. The first flash memory product came out in 1988 but it did not take off until the late 1990’s because it could not be reliable produced. NOR and NAND memory is named after the arrangement of the cells in the memory array. NOR based flash memory benefits from having very fast burst read times but slower write times. Due to the structure of NOR memory programs stored in NOR based memory can be executed without being loaded into RAM first. NAND flash memory has a very large storage capacity and can read and write large files relatively fast. NAND is more suited for storage while NOR memory is better suited for direct program execution such as in CMOS chips'''

HDDs use a block system, in which the kernel specifies which blocks to read and write. When using a flash drive, the blocks are emulated and mapped to a physical memory address. It does through what is called a "Transition Layer".

==Traditionally Optimized File Systems==

Since the kernel asks for a block number, a conventional hard disk drive (HDD) file-system is not optimized to work with flash memory. The reason for this is that conventional hard-disks have different constraints from those of flash memory - their primary problem is to reduce seeking time, while the primary problem when working with flash memory is to erase in a minimal and balanced way.

The most consuming process for an HDD is seeking data by relocating the read-head and spinning the magnetic disk. A traditional file system optimizes the way it stores data by placing related blocks close-by on the disk in order to minimize mechanical movement within the HDD. One of the great advantages of flash memory, which accounts for its fast read speed, is that there is no need to seek data physically. This is also why defragmentaion, a procedure used by HDDs to put files into more convenient configurations and thus minimize seeking times, loses its purpose in a flash memory context. Indeed, the unnecessary erasures that it entails are both inefficient and harmful for a flash memory unit.

This comes directly out of flash memory's aforementioned constraints: the slow block-sized erasures and the limited number of erase-cycles. Because of these, a flash optimal file system needs to minimize its erase operations and also to spread out its erasures in such a way as to avoid the formation of hot-spots: sections of memory which have undergone a disproportionately high number of erasures and are thus in danger of burning out. This process of spreading out data is referred to as "wear leveling". To minimize hotspots, a system using flash memory would have to write new data to empty memory blocks. This method would also call for some sort of garbage collection to conduct necessary erasure operations while the system is idle. It makes better sense to do these at this time because of the slow nature of erasures in the flash memory context. Of course, there is no such feature in a traditional HDD file-system.

'''More on this later'''

==Flash Optimized File Systems==

The process of "wear leveling" [WHAT IS THIS?] is achieved through a Log-based File System, often referred to as the Flash Transitional Layer(FTL). Essentially, the drive stores a log that keeps track of how many times each erase sector has been invalidated (or erased). The translational layer has a translation table, where each physical memory address is associated with an emulated block sectors. This allows a traditional file system that uses block sectors to be used on the flash drive. Each block has a flag which keeps track of its state. When a block is being written to, the FTL marks the blocks needed as ''allocated''. This prevents other data being written to the block that has already been allocated. The FTL then goes on to write the data in the allocated blocks. Once it completes the transaction, the system updates the allocated blocks to ''pre-valid''. Once that is completed, the drive marks the invalidated blocks to ''invalid'', while marking the newly written block as ''valid''. This entire flagging process is to ensure that the newly allocated blocks are never mixed up with the invalidated blocks.

===Banks===
The FTL organizes data using structures called banks. When the FTL gets a request to write something to memory, it uses a bank list to determine which area of the drive should be used. Essentially a bank is a group of sequential addresses, that keeps track of when it was last updated using timestamps. The FTL will only write to that bank, and once there is not enough space to write anymore, it switches out the current bank for the one with the most available space. When cleaning up the bank, the system puts it into what is called the Cleaning Bank List and removes it from the Bank List, thus avoiding any chance of some data being written to that bank while something is being erased.

===Cleaner===
When the FTL realizes that there is not enough room to write new data onto the drive, it runs a garbage collection routine. This routine selects a segment to be cleaned, copies all of the valid data into a new segment, then erases everything in the old segment. This frees up the otherwise useless invalidated blocks and by not erasing every block as soon as it becomes invalidated, it saves on the amount of times that the expensive erase operation is called. The kernel can preemptively clean the drive when the system is idle.

=Conclusion=

In this way, thanks to its [WHATEVER MAKES LOG FSs ACTUALLY GOOD AT DEALING WITH FLASH], the log-based file-system is far better suited to working with flash memory than a traditional HDD file system. The latter is utterly unfit for this task due to its placing primacy on the minimization of seeks rather than on the minimization and management of erasures. Dealing smartly with erasures is extremely important for a flash memory file system, as that memory type's particular weaknesses, the limited number of erasure cycles, the necessity to erase by the block and the relative slowness of the erasures themselves, all have to do with erasing. A good flash memory file system must therefore be built with the aim of making the best of these weaknesses and this is precisely the reason why older disk-based file systems are not suitable for flash memory while log-based file systems are. [INSPIRATIONAL LAST WORDS]

=Questions=

=References=

[1] Kim, Han-joon; Lee, Sang-goo. ''A New Flash Memory Management for Flash Storage System''. ''IEEExplore''. Dept. of Comput. Sci., Seoul Nat. Univ., 06 Aug 2002. <http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=812717&tag=1#>

[2] Smith, Lance. ''NAND Flash Solid State Storage Performance and Capability''. ''Flash Memory Summit''. SNIA Education Committee, 18 Aug 2009. <http://www.flashmemorysummit.com/English/Collaterals/Proceedings/2009/20090812_T1B_Smith.pdf>

[3] Chang, LiPin. ''On Efficient Wear Leveling for Large-Scale Flash-Memory Storage Systems''. ''Association for Computing Machinery (ACM)''. Dept. of Comput. Sci.,Nat. ChiaoTung Univ., 15 Mar 2007. <http://portal.acm.org/citation.cfm?id=1244248>

[4] Nath, Suman; Gibbons, Phillip. ''Online maintenance of very large random samples on flash storage''. ''Association for Computing Machinery (ACM)''. The VLDB Journal, 27 Jul 2007. <http://portal.acm.org/citation.cfm?id=1731355>

[5] Lim, Seung-Ho; Park; Kyu-Ho. ''An Efficient NAND Flash File System for Flash Memory Storage''. ''CORE Laboratory''. IEEE Transactions On Computers, Jul 2006. <http://vlsi.kaist.ac.kr/paper_list/2006_TC_CFFS.pdf>

[6] ''NAND vs. NOR Flash Memory Technology Overview''. ''RMG and Associates''. Toshiba America, accessed 14 Oct 2010. <http://maltiel-consulting.com/NAND_vs_NOR_Flash_Memory_Technology_Overview_Read_Write_Erase_speed_for_SLC_MLC_semiconductor_consulting_expert.pdf>

[7] Bez, Roberto; Camerlenghi, Emilio; Modelli, Alberto; Visconti, Angelo. ''Introduction to Flash Memory''. ''IEEExplore''. STMicroelectronics, 21 May 2003. <http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1199079&tag=1>

[8] Kawaguchi, Atsuo; Nishioka, Shingo; Motoda Hiroshi. ''A Flash-Memory Based File System''. ''CiteSeerX'' Advanced Research laboratory, Hitachi, Ltd., 1995. <http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.160.5142>

[9] Rosenblum, Mendel; Ouserhout, John. ''The Design and Implementation of a Log-structured File System''. ''Association for Computing Machinery (ACM)''. University of California at Berkeley, Feb 1992. <http://portal.acm.org/citation.cfm?id=146943&coll=GUIDE&dl=GUIDE&CFID=108397378&CFTOKEN=72657973&ret=1#Fulltext>

[10] Shimpi, Anand. ''Intel X25-M SSD: Intel Delivers One of the World's Fastest Drives''. ''AnAndTech''. AnAndTech, 8 Sep 2008. <http://www.anandtech.com/show/2614>

[11] Shimpi, Anand. ''The SSD Relapse: Understanding and Choosing the Best SSD''. ''AnAndTech''. AnAndTech, 30 Aug 2009. <http://www.anandtech.com/show/2829>

[12] Shimpi, Anand. ''The SSD Anthology: Understanding SSDs and New Drives from OCZ''. ''AnAndTech''. AnAndTech, 18 Mar 2009. <http://www.anandtech.com/show/2738>

[13] Corbet, Jonathan. ''Solid-State Storage Devices and the Block Layer''. ''Linux Weekly News''. Linux Weekly News, 4 Oct 2010. <http://lwn.net/Articles/408428/>

[14] Woodhouse, David. ''JFFS : The Journalling Flash File System''. ''CiteSeerX''. Red Hat, Inc, Accessed 14 Oct 2010. <http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.128.6156&rep=rep1&type=pdf>

[15] Agrawal, Nitin; Prabhakaran, Vijayan; Wobber, Ted; Davis, John; Manasse, Mark. Panigrahy, Rina. ''Design Tradeoffs for SSD Performance''. ''Association for Computing Machinery (ACM)'', USENIX 2008 Annual Technical Conference, 2008. <http://portal.acm.org/citation.cfm?id=1404014.1404019>

[16] Lee, Sang-Won, et al. ''A Log Buffer-Based Flash Translation Layer Using Fully-Associative Sector Translation''. ''Association for Computing Machinery (ACM)''. ACM Transactions on Embedded Computing Systems (TECS), Jul 2007. <http://portal.acm.org/citation.cfm?id=1275990>

[17] ''Reach New Heights in Computing Performance''. ''Micron Technology Inc''. Micro Technology Inc, Accessed 14 Oct 2010. <http://www.micron.com/products/solid_state_storage/client_ssd.html>

[18] ''Flash Memories.'' 1 ed. New York: Springer, 1999. Print.

[19] ''Nonvolatile Memory Technologies with Emphasis on Flash: A Comprehensive Guide to Understanding and Using Flash Memory Devices''. ''IEEE Press Series on Microelectronic Systems''. New York: Wiley-Ieee Press, 2008. Print.

[20] ''Nonvolatile Semiconductor Memory Technology: A Comprehensive Guide to Understanding and Using NVSM Devices''. ''IEEE Press Series on Microelectronic Systems''. New York: Wiley-Ieee Press, 1997. Print.

=External links=

Relevant Wikipedia articles: [http://en.wikipedia.org/wiki/Flash_Memory Flash Memory], [http://en.wikipedia.org/wiki/LogFS LogFS], [http://en.wikipedia.org/wiki/Hard_disk Hard Disk Drives], [http://en.wikipedia.org/wiki/Wear_leveling Wear Leveling], [http://en.wikipedia.org/wiki/Hot_spot_%28computer_science%29 Hot Spots], [http://en.wikipedia.org/wiki/Solid-state_drive Sold State Drive].

=old=

Hey guys,

This is what I've got so far... mostly based on wikipedia:

Flash memory has two limitations: it can only be erased in blocks and and it wears out after a certain number of erase cycles. Furthermore, a particular kind of Flash memory (NAND) is not able to provide random access.
As a result of these Flash based file-systems cannot be handled in the same way as disk-based file systems. Here are a few of the key differences:

- Because memory must be erased in blocks, its erasure tends to take up time. Consequently, it is necessary to time the erasures in a way so as not to interfere with the efficiency of the system’s other operations. This is is not a real concern with disk-based file-systems.
- A disk file-system needs to minimize the seeking time, but Flash file-system does not concern itself with this as it doesn’t have a disk.
- A flash system tries to distribute memory in such a way so as not to make a particular block of memory subject to a disproportionally large number of erasures. The purpose of this is to keep the block from wearing out prematurely. The result of it is that memory needs to be distributed differently than in a disk based file-system.
Log-sturctured file systems are thus best suited to dealing with flash memory (they apparently do all of the above things).

For the essay form, I'm thinking of doing a section about traditional hard-disk systems, another about flash-memory and a third about flash systems. At this point, I am imagining the thesis as something like, "Flash systems require a fundamentally different system architecture than disk-based systems due to their need to adapt to the constraints inherent in flash memory: specifically, due to that memory's limited life-span and block-based erasures." The argument would then talk about how these two differences directly lead to a new FS approach.

That's how I see it at the moment. Honestly, I don't like doing research about this kind of stuff, so my data isn't very deep. That said, if you guys could find more info and summarize it, I'm pretty sure that I could synthesize it all into a coherent essay.

Fedor

COMP 3000 Essay 1 2010 Question 10

2010-10-15T03:25:41Z

Gsmith6: /* Flash Memory */

=Question=

How do the constraints of flash storage affect the design of flash-optimized file systems? Explain by contrasting with hard disk-based file systems.

=Answer=
First introduced in the late 80s, Flash-memory is a light, energy-independent, compact, shock-resistant and efficiently readable type of storage. Because of the particular limitations of this kind of memory, flash file systems require a fundamentally different system architecture than disk-based file-systems: these systems need to be designed in light of flash-memory’s limited number of erase-cycles and its need to conduct erasures one entire block at a time. These constraints are a direct result of the same design that gives flash its advantages with regard to [ TO WHAT?] as both are due to [TO WHAT?] . Thus, a typical disk-based file-system is not suitable for working with flash memory as it erases far too frequently and indiscriminately while being simultaneously optimized for other constraints that do not affect flash memory. This means that a different solution is necessary and that solution is the log-based file-system which is far better suited to working with flash memory because it optimizes erasures by [WHAT?].

==Flash Memory==
Flash memory is non-volatile(meaning digital storage that does not require power to retain its memory) storage space that has become more popular recently due to its fast fetch times. There are two basic forms of the flash storage system, NOR and NAND. Each type has its advantages and disadvantages. NOR has the fastest read times, but is much slower at writing. NAND on the other hand has much more capacity, faster write times, is less expensive, and has a much longer life expectancy.[2]

More and more people use flash memory, with many sizes of drives, ranging from a few hundred megabyte USB key, to a few terabyte internal solid-state drive(SSD). Two main reasons for this movement are because of flash's extremely fast read times, and its falling price. A typical flash drive has read speeds of up to 14 times faster than a hard disk drive (HDD).[17]

Although flash drives are exponentially faster than HDDs, they still have not become the main source of data management. The reason for this is because HDDs are simply much cheaper, and flash drives still have many faults. The most critical fault is that each block in flash memory can only be erased approximately 100,000 times.[14] This poses a problem because when modifying a file, even if its a single bit, the entire block must be erased, and rewritten. This erase/rewrite slows down the write operation considerably, making it actually slower to write a file to flash than an HDD.[8]

'''1. What flash storage is, why its good but also why it must have the problems that it does (the assumption is that it must have them, why would it otherwise?) [don't know much about this just now... basics include that there is NOR (reads slightly faster)and NAND (holds more, writes faster, erases much faster, lasts about ten times longer) flash with NAND being especially popular for storage (what's NOR good for?). Here, we'd ideally want to talk about why flash was invented (supposed as an alternative to slow ROM), why it was suitable for that, and how it works on a technical level. Then, we'd want to mention why this technical functionality was innovative and useful but also why it came with two serious set-backs: having a limited-number of re-write cycles and needing to erase a block at a time.'''
'''Either way, Flash storage affords far faster fetch times than the traditional platter-based HDD, and stability of information in a sense. Where the data is not actually stored, but reprogrammed, in a sense, the data is more secure and is less likely to be erased easily. On that note, in order to flip a single bit, that entire block will need to be erased, then reprogrammed. In an 'old' HDD, let's say, When the HDD fails at the end of its life cycle, your data is gone. (unless you're willing to shell out $200/hr to have it recovered, yes I've seen companies in Ottawa that do this.) In a flash HDD, when it reaches the end of its life, it merely becomes read-only. Bugger for Databases, but useful for technical notes and archives, let's say. With today's modern gaming computers, Flash memory can be good on quick load times, however with limited read-writes, it could afford better use to things that are not updated as frequently. I.e... Well I don't have a better example than a webserver hosting a company's CSS and scripts. ~Source: Years in the 'biz
Flash memory started out as a replacement for EPROMs. At the time EPROMs needed a UV photoemission to be erased while flash memory could be erased electronically. The first flash memory product came out in 1988 but it did not take off until the late 1990’s because it could not be reliable produced. NOR and NAND memory is named after the arrangement of the cells in the memory array. NOR based flash memory benefits from having very fast burst read times but slower write times. Due to the structure of NOR memory programs stored in NOR based memory can be executed without being loaded into RAM first. NAND flash memory has a very large storage capacity and can read and write large files relatively fast. NAND is more suited for storage while NOR memory is better suited for direct program execution such as in CMOS chips'''

HDDs use a block system, in which the kernel specifies which blocks to read and write. When using a flash drive, the blocks are emulated and mapped to a physical memory address. It does through what is called a "Transition Layer".

==Traditionally Optimized File Systems==

Since the kernel asks for a block number, a conventional hard disk drive (HDD) file-system is not optimized to work with flash memory. The reason for this is that conventional hard-disks have different constraints from those of flash memory - their primary problem is to reduce seeking time, while the primary problem when working with flash memory is to erase in a minimal and balanced way.

The most consuming process for an HDD is seeking data by relocating the read-head and spinning the magnetic disk. A traditional file system optimizes the way it stores data by placing related blocks close-by on the disk in order to minimize mechanical movement within the HDD. One of the great advantages of flash memory, which accounts for its fast read speed, is that there is no need to seek data physically. This is also why defragmentaion, a procedure used by HDDs to put files into more convenient configurations and thus minimize seeking times, loses its purpose in a flash memory context. Indeed, the unnecessary erasures that it entails are both inefficient and harmful for a flash memory unit.

This comes directly out of flash memory's aforementioned constraints: the slow block-sized erasures and the limited number of erase-cycles. Because of these, a flash optimal file system needs to minimize its erase operations and also to spread out its erasures in such a way as to avoid the formation of hot-spots: sections of memory which have undergone a disproportionately high number of erasures and are thus in danger of burning out. This process of spreading out data is referred to as "wear leveling". To minimize hotspots, a system using flash memory would have to write new data to empty memory blocks. This method would also call for some sort of garbage collection to conduct necessary erasure operations while the system is idle. It makes better sense to do these at this time because of the slow nature of erasures in the flash memory context. Of course, there is no such feature in a traditional HDD file-system.

'''More on this later'''

==Flash Optimized File Systems==

The process of "wear leveling" [WHAT IS THIS?] is achieved through a Log-based File System, often referred to as the Flash Transitional Layer(FTL). Essentially, the drive stores a log that keeps track of how many times each erase sector has been invalidated (or erased). The translational layer has a translation table, where each physical memory address is associated with an emulated block sectors. This allows a traditional file system that uses block sectors to be used on the flash drive. Each block has a flag which keeps track of its state. When a block is being written to, the FTL marks the blocks needed as ''allocated''. This prevents other data being written to the block that has already been allocated. The FTL then goes on to write the data in the allocated blocks. Once it completes the transaction, the system updates the allocated blocks to ''pre-valid''. Once that is completed, the drive marks the invalidated blocks to ''invalid'', while marking the newly written block as ''valid''. This entire flagging process is to ensure that the newly allocated blocks are never mixed up with the invalidated blocks.

===Banks===
The FTL organizes data using structures called banks. When the FTL gets a request to write something to memory, it uses a bank list to determine which area of the drive should be used. Essentially a bank is a group of sequential addresses, that keeps track of when it was last updated using timestamps. The FTL will only write to that bank, and once there is not enough space to write anymore, it switches out the current bank for the one with the most available space. When cleaning up the bank, the system puts it into what is called the Cleaning Bank List and removes it from the Bank List, thus avoiding any chance of some data being written to that bank while something is being erased.

===Cleaner===
When the FTL realizes that there is not enough room to write new data onto the drive, it runs a garbage collection routine. This routine selects a segment to be cleaned, copies all of the valid data into a new segment, then erases everything in the old segment. This frees up the otherwise useless invalidated blocks and by not erasing every block as soon as it becomes invalidated, it saves on the amount of times that the expensive erase operation is called. The kernel can preemptively clean the drive when the system is idle.

=Conclusion=

In this way, thanks to its [WHATEVER MAKES LOG FSs ACTUALLY GOOD AT DEALING WITH FLASH], the log-based file-system is far better suited to working with flash memory than a traditional HDD file system. The latter is utterly unfit for this task due to its placing primacy on the minimization of seeks rather than on the minimization and management of erasures. Dealing smartly with erasures is extremely important for a flash memory file system, as that memory type's particular weaknesses, the limited number of erasure cycles, the necessity to erase by the block and the relative slowness of the erasures themselves, all have to do with erasing. A good flash memory file system must therefore be built with the aim of making the best of these weaknesses and this is precisely the reason why older disk-based file systems are not suitable for flash memory while log-based file systems are. [INSPIRATIONAL LAST WORDS]

=Questions=

=References=

[1] Kim, Han-joon; Lee, Sang-goo. ''A New Flash Memory Management for Flash Storage System''. ''IEEExplore''. Dept. of Comput. Sci., Seoul Nat. Univ., 06 Aug 2002. <http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=812717&tag=1#>

[2] Smith, Lance. ''NAND Flash Solid State Storage Performance and Capability''. ''Flash Memory Summit''. SNIA Education Committee, 18 Aug 2009. <http://www.flashmemorysummit.com/English/Collaterals/Proceedings/2009/20090812_T1B_Smith.pdf>

[3] Chang, LiPin. ''On Efficient Wear Leveling for Large-Scale Flash-Memory Storage Systems''. ''Association for Computing Machinery (ACM)''. Dept. of Comput. Sci.,Nat. ChiaoTung Univ., 15 Mar 2007. <http://portal.acm.org/citation.cfm?id=1244248>

[4] Nath, Suman; Gibbons, Phillip. ''Online maintenance of very large random samples on flash storage''. ''Association for Computing Machinery (ACM)''. The VLDB Journal, 27 Jul 2007. <http://portal.acm.org/citation.cfm?id=1731355>

[5] Lim, Seung-Ho; Park; Kyu-Ho. ''An Efficient NAND Flash File System for Flash Memory Storage''. ''CORE Laboratory''. IEEE Transactions On Computers, Jul 2006. <http://vlsi.kaist.ac.kr/paper_list/2006_TC_CFFS.pdf>

[6] ''NAND vs. NOR Flash Memory Technology Overview''. ''RMG and Associates''. Toshiba America, accessed 14 Oct 2010. <http://maltiel-consulting.com/NAND_vs_NOR_Flash_Memory_Technology_Overview_Read_Write_Erase_speed_for_SLC_MLC_semiconductor_consulting_expert.pdf>

[7] Bez, Roberto; Camerlenghi, Emilio; Modelli, Alberto; Visconti, Angelo. ''Introduction to Flash Memory''. ''IEEExplore''. STMicroelectronics, 21 May 2003. <http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1199079&tag=1>

[8] Kawaguchi, Atsuo; Nishioka, Shingo; Motoda Hiroshi. ''A Flash-Memory Based File System''. ''CiteSeerX'' Advanced Research laboratory, Hitachi, Ltd., 1995. <http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.160.5142>

[9] Rosenblum, Mendel; Ouserhout, John. ''The Design and Implementation of a Log-structured File System''. ''Association for Computing Machinery (ACM)''. University of California at Berkeley, Feb 1992. <http://portal.acm.org/citation.cfm?id=146943&coll=GUIDE&dl=GUIDE&CFID=108397378&CFTOKEN=72657973&ret=1#Fulltext>

[10] Shimpi, Anand. ''Intel X25-M SSD: Intel Delivers One of the World's Fastest Drives''. ''AnAndTech''. AnAndTech, 8 Sep 2008. <http://www.anandtech.com/show/2614>

[11] Shimpi, Anand. ''The SSD Relapse: Understanding and Choosing the Best SSD''. ''AnAndTech''. AnAndTech, 30 Aug 2009. <http://www.anandtech.com/show/2829>

[12] Shimpi, Anand. ''The SSD Anthology: Understanding SSDs and New Drives from OCZ''. ''AnAndTech''. AnAndTech, 18 Mar 2009. <http://www.anandtech.com/show/2738>

[13] Corbet, Jonathan. ''Solid-State Storage Devices and the Block Layer''. ''Linux Weekly News''. Linux Weekly News, 4 Oct 2010. <http://lwn.net/Articles/408428/>

[14] Woodhouse, David. ''JFFS : The Journalling Flash File System''. ''CiteSeerX''. Red Hat, Inc, Accessed 14 Oct 2010. <http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.128.6156&rep=rep1&type=pdf>

[15] Agrawal, Nitin; Prabhakaran, Vijayan; Wobber, Ted; Davis, John; Manasse, Mark. Panigrahy, Rina. ''Design Tradeoffs for SSD Performance''. ''Association for Computing Machinery (ACM)'', USENIX 2008 Annual Technical Conference, 2008. <http://portal.acm.org/citation.cfm?id=1404014.1404019>

[16] Lee, Sang-Won, et al. ''A Log Buffer-Based Flash Translation Layer Using Fully-Associative Sector Translation''. ''Association for Computing Machinery (ACM)''. ACM Transactions on Embedded Computing Systems (TECS), Jul 2007. <http://portal.acm.org/citation.cfm?id=1275990>

[17] ''Reach New Heights in Computing Performance''. ''Micron Technology Inc''. Micro Technology Inc, Accessed 14 Oct 2010. <http://www.micron.com/products/solid_state_storage/client_ssd.html>

[18] ''Flash Memories.'' 1 ed. New York: Springer, 1999. Print.

[19] ''Nonvolatile Memory Technologies with Emphasis on Flash: A Comprehensive Guide to Understanding and Using Flash Memory Devices''. ''IEEE Press Series on Microelectronic Systems''. New York: Wiley-Ieee Press, 2008. Print.

[20] ''Nonvolatile Semiconductor Memory Technology: A Comprehensive Guide to Understanding and Using NVSM Devices''. ''IEEE Press Series on Microelectronic Systems''. New York: Wiley-Ieee Press, 1997. Print.

=External links=

Relevant Wikipedia articles: [http://en.wikipedia.org/wiki/Flash_Memory Flash Memory], [http://en.wikipedia.org/wiki/LogFS LogFS], [http://en.wikipedia.org/wiki/Hard_disk Hard Disk Drives], [http://en.wikipedia.org/wiki/Wear_leveling Wear Leveling], [http://en.wikipedia.org/wiki/Hot_spot_%28computer_science%29 Hot Spots], [http://en.wikipedia.org/wiki/Solid-state_drive Sold State Drive].

=old=

Hey guys,

This is what I've got so far... mostly based on wikipedia:

Flash memory has two limitations: it can only be erased in blocks and and it wears out after a certain number of erase cycles. Furthermore, a particular kind of Flash memory (NAND) is not able to provide random access.
As a result of these Flash based file-systems cannot be handled in the same way as disk-based file systems. Here are a few of the key differences:

- Because memory must be erased in blocks, its erasure tends to take up time. Consequently, it is necessary to time the erasures in a way so as not to interfere with the efficiency of the system’s other operations. This is is not a real concern with disk-based file-systems.
- A disk file-system needs to minimize the seeking time, but Flash file-system does not concern itself with this as it doesn’t have a disk.
- A flash system tries to distribute memory in such a way so as not to make a particular block of memory subject to a disproportionally large number of erasures. The purpose of this is to keep the block from wearing out prematurely. The result of it is that memory needs to be distributed differently than in a disk based file-system.
Log-sturctured file systems are thus best suited to dealing with flash memory (they apparently do all of the above things).

For the essay form, I'm thinking of doing a section about traditional hard-disk systems, another about flash-memory and a third about flash systems. At this point, I am imagining the thesis as something like, "Flash systems require a fundamentally different system architecture than disk-based systems due to their need to adapt to the constraints inherent in flash memory: specifically, due to that memory's limited life-span and block-based erasures." The argument would then talk about how these two differences directly lead to a new FS approach.

That's how I see it at the moment. Honestly, I don't like doing research about this kind of stuff, so my data isn't very deep. That said, if you guys could find more info and summarize it, I'm pretty sure that I could synthesize it all into a coherent essay.

Fedor

COMP 3000 Essay 1 2010 Question 10

2010-10-15T03:24:05Z

Gsmith6: /* Flash Memory */

=Question=

How do the constraints of flash storage affect the design of flash-optimized file systems? Explain by contrasting with hard disk-based file systems.

=Answer=
First introduced in the late 80s, Flash-memory is a light, energy-independent, compact, shock-resistant and efficiently readable type of storage. Because of the particular limitations of this kind of memory, flash file systems require a fundamentally different system architecture than disk-based file-systems: these systems need to be designed in light of flash-memory’s limited number of erase-cycles and its need to conduct erasures one entire block at a time. These constraints are a direct result of the same design that gives flash its advantages with regard to [ TO WHAT?] as both are due to [TO WHAT?] . Thus, a typical disk-based file-system is not suitable for working with flash memory as it erases far too frequently and indiscriminately while being simultaneously optimized for other constraints that do not affect flash memory. This means that a different solution is necessary and that solution is the log-based file-system which is far better suited to working with flash memory because it optimizes erasures by [WHAT?].

==Flash Memory==
Flash memory is non-volatile(meaning digital storage that does not require power to retain its memory) storage space that has become more popular recently due to its fast fetch times. There are two basic forms of the flash storage system, NOR and NAND. Each type has its advantages and disadvantages. NOR has the fastest read times, but is much slower at writing. NAND on the other hand has much more capacity, faster write times, is less expensive, and has a much longer life expectancy.[2]

More and more people use flash memory, with many sizes of drives, ranging from a few hundred megabyte USB key, to a few terabyte internal solid-state drive(SSD). Two main reasons for this movement are because of flash's extremely fast read times, and its falling price. A typical flash drive has read speeds of up to 14 times faster than a hard disk drive (HDD).[17]

Although flash drives are exponentially faster than HDDs, they still have not become the main source of data management. The reason for this is because HDDs are simply much cheaper, and flash drives still have many faults. The most critical fault is that each block in flash memory can only be erased approximately 100,000 times.[14] This poses a problem because when modifying a file, even if its a single bit, the entire block must be erased, and rewritten. This erase/rewrite slows down the write operation considerably, making it actually slower to write a file to flash than an HDD.[8]

'''1. What flash storage is, why its good but also why it must have the problems that it does (the assumption is that it must have them, why would it otherwise?) [don't know much about this just now... basics include that there is NOR (reads slightly faster)and NAND (holds more, writes faster, erases much faster, lasts about ten times longer) flash with NAND being especially popular for storage (what's NOR good for?). Here, we'd ideally want to talk about why flash was invented (supposed as an alternative to slow ROM), why it was suitable for that, and how it works on a technical level. Then, we'd want to mention why this technical functionality was innovative and useful but also why it came with two serious set-backs: having a limited-number of re-write cycles and needing to erase a block at a time.
Either way, Flash storage affords far faster fetch times than the traditional platter-based HDD, and stability of information in a sense. Where the data is not actually stored, but reprogrammed, in a sense, the data is more secure and is less likely to be erased easily. On that note, in order to flip a single bit, that entire block will need to be erased, then reprogrammed. In an 'old' HDD, let's say, When the HDD fails at the end of its life cycle, your data is gone. (unless you're willing to shell out $200/hr to have it recovered, yes I've seen companies in Ottawa that do this.) In a flash HDD, when it reaches the end of its life, it merely becomes read-only. Bugger for Databases, but useful for technical notes and archives, let's say. With today's modern gaming computers, Flash memory can be good on quick load times, however with limited read-writes, it could afford better use to things that are not updated as frequently. I.e... Well I don't have a better example than a webserver hosting a company's CSS and scripts. ~Source: Years in the 'biz
Flash memory started out as a replacement for EPROMs. At the time EPROMs needed a UV photoemission to be erased while flash memory could be erased electronically. The first flash memory product came out in 1988 but it did not take off until the late 1990’s because it could not be reliable produced. NOR and NAND memory is named after the arrangement of the cells in the memory array. NOR based flash memory benefits from having very fast burst read times but slower write times. Due to the structure of NOR memory programs stored in NOR based memory can be executed without being loaded into RAM first. NAND flash memory has a very large storage capacity and can read and write large files relatively fast. NAND is more suited for storage while NOR memory is better suited for direct program execution such as in CMOS chips'''

HDDs use a block system, in which the kernel specifies which blocks to read and write. When using a flash drive, the blocks are emulated and mapped to a physical memory address. It does through what is called a "Transition Layer".

==Traditionally Optimized File Systems==

Since the kernel asks for a block number, a conventional hard disk drive (HDD) file-system is not optimized to work with flash memory. The reason for this is that conventional hard-disks have different constraints from those of flash memory - their primary problem is to reduce seeking time, while the primary problem when working with flash memory is to erase in a minimal and balanced way.

The most consuming process for an HDD is seeking data by relocating the read-head and spinning the magnetic disk. A traditional file system optimizes the way it stores data by placing related blocks close-by on the disk in order to minimize mechanical movement within the HDD. One of the great advantages of flash memory, which accounts for its fast read speed, is that there is no need to seek data physically. This is also why defragmentaion, a procedure used by HDDs to put files into more convenient configurations and thus minimize seeking times, loses its purpose in a flash memory context. Indeed, the unnecessary erasures that it entails are both inefficient and harmful for a flash memory unit.

This comes directly out of flash memory's aforementioned constraints: the slow block-sized erasures and the limited number of erase-cycles. Because of these, a flash optimal file system needs to minimize its erase operations and also to spread out its erasures in such a way as to avoid the formation of hot-spots: sections of memory which have undergone a disproportionately high number of erasures and are thus in danger of burning out. This process of spreading out data is referred to as "wear leveling". To minimize hotspots, a system using flash memory would have to write new data to empty memory blocks. This method would also call for some sort of garbage collection to conduct necessary erasure operations while the system is idle. It makes better sense to do these at this time because of the slow nature of erasures in the flash memory context. Of course, there is no such feature in a traditional HDD file-system.

'''More on this later'''

==Flash Optimized File Systems==

The process of "wear leveling" [WHAT IS THIS?] is achieved through a Log-based File System, often referred to as the Flash Transitional Layer(FTL). Essentially, the drive stores a log that keeps track of how many times each erase sector has been invalidated (or erased). The translational layer has a translation table, where each physical memory address is associated with an emulated block sectors. This allows a traditional file system that uses block sectors to be used on the flash drive. Each block has a flag which keeps track of its state. When a block is being written to, the FTL marks the blocks needed as ''allocated''. This prevents other data being written to the block that has already been allocated. The FTL then goes on to write the data in the allocated blocks. Once it completes the transaction, the system updates the allocated blocks to ''pre-valid''. Once that is completed, the drive marks the invalidated blocks to ''invalid'', while marking the newly written block as ''valid''. This entire flagging process is to ensure that the newly allocated blocks are never mixed up with the invalidated blocks.

===Banks===
The FTL organizes data using structures called banks. When the FTL gets a request to write something to memory, it uses a bank list to determine which area of the drive should be used. Essentially a bank is a group of sequential addresses, that keeps track of when it was last updated using timestamps. The FTL will only write to that bank, and once there is not enough space to write anymore, it switches out the current bank for the one with the most available space. When cleaning up the bank, the system puts it into what is called the Cleaning Bank List and removes it from the Bank List, thus avoiding any chance of some data being written to that bank while something is being erased.

===Cleaner===
When the FTL realizes that there is not enough room to write new data onto the drive, it runs a garbage collection routine. This routine selects a segment to be cleaned, copies all of the valid data into a new segment, then erases everything in the old segment. This frees up the otherwise useless invalidated blocks and by not erasing every block as soon as it becomes invalidated, it saves on the amount of times that the expensive erase operation is called. The kernel can preemptively clean the drive when the system is idle.

=Conclusion=

In this way, thanks to its [WHATEVER MAKES LOG FSs ACTUALLY GOOD AT DEALING WITH FLASH], the log-based file-system is far better suited to working with flash memory than a traditional HDD file system. The latter is utterly unfit for this task due to its placing primacy on the minimization of seeks rather than on the minimization and management of erasures. Dealing smartly with erasures is extremely important for a flash memory file system, as that memory type's particular weaknesses, the limited number of erasure cycles, the necessity to erase by the block and the relative slowness of the erasures themselves, all have to do with erasing. A good flash memory file system must therefore be built with the aim of making the best of these weaknesses and this is precisely the reason why older disk-based file systems are not suitable for flash memory while log-based file systems are. [INSPIRATIONAL LAST WORDS]

=Questions=

=References=

[1] Kim, Han-joon; Lee, Sang-goo. ''A New Flash Memory Management for Flash Storage System''. ''IEEExplore''. Dept. of Comput. Sci., Seoul Nat. Univ., 06 Aug 2002. <http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=812717&tag=1#>

[2] Smith, Lance. ''NAND Flash Solid State Storage Performance and Capability''. ''Flash Memory Summit''. SNIA Education Committee, 18 Aug 2009. <http://www.flashmemorysummit.com/English/Collaterals/Proceedings/2009/20090812_T1B_Smith.pdf>

[3] Chang, LiPin. ''On Efficient Wear Leveling for Large-Scale Flash-Memory Storage Systems''. ''Association for Computing Machinery (ACM)''. Dept. of Comput. Sci.,Nat. ChiaoTung Univ., 15 Mar 2007. <http://portal.acm.org/citation.cfm?id=1244248>

[4] Nath, Suman; Gibbons, Phillip. ''Online maintenance of very large random samples on flash storage''. ''Association for Computing Machinery (ACM)''. The VLDB Journal, 27 Jul 2007. <http://portal.acm.org/citation.cfm?id=1731355>

[5] Lim, Seung-Ho; Park; Kyu-Ho. ''An Efficient NAND Flash File System for Flash Memory Storage''. ''CORE Laboratory''. IEEE Transactions On Computers, Jul 2006. <http://vlsi.kaist.ac.kr/paper_list/2006_TC_CFFS.pdf>

[6] ''NAND vs. NOR Flash Memory Technology Overview''. ''RMG and Associates''. Toshiba America, accessed 14 Oct 2010. <http://maltiel-consulting.com/NAND_vs_NOR_Flash_Memory_Technology_Overview_Read_Write_Erase_speed_for_SLC_MLC_semiconductor_consulting_expert.pdf>

[7] Bez, Roberto; Camerlenghi, Emilio; Modelli, Alberto; Visconti, Angelo. ''Introduction to Flash Memory''. ''IEEExplore''. STMicroelectronics, 21 May 2003. <http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1199079&tag=1>

[8] Kawaguchi, Atsuo; Nishioka, Shingo; Motoda Hiroshi. ''A Flash-Memory Based File System''. ''CiteSeerX'' Advanced Research laboratory, Hitachi, Ltd., 1995. <http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.160.5142>

[9] Rosenblum, Mendel; Ouserhout, John. ''The Design and Implementation of a Log-structured File System''. ''Association for Computing Machinery (ACM)''. University of California at Berkeley, Feb 1992. <http://portal.acm.org/citation.cfm?id=146943&coll=GUIDE&dl=GUIDE&CFID=108397378&CFTOKEN=72657973&ret=1#Fulltext>

[10] Shimpi, Anand. ''Intel X25-M SSD: Intel Delivers One of the World's Fastest Drives''. ''AnAndTech''. AnAndTech, 8 Sep 2008. <http://www.anandtech.com/show/2614>

[11] Shimpi, Anand. ''The SSD Relapse: Understanding and Choosing the Best SSD''. ''AnAndTech''. AnAndTech, 30 Aug 2009. <http://www.anandtech.com/show/2829>

[12] Shimpi, Anand. ''The SSD Anthology: Understanding SSDs and New Drives from OCZ''. ''AnAndTech''. AnAndTech, 18 Mar 2009. <http://www.anandtech.com/show/2738>

[13] Corbet, Jonathan. ''Solid-State Storage Devices and the Block Layer''. ''Linux Weekly News''. Linux Weekly News, 4 Oct 2010. <http://lwn.net/Articles/408428/>

[14] Woodhouse, David. ''JFFS : The Journalling Flash File System''. ''CiteSeerX''. Red Hat, Inc, Accessed 14 Oct 2010. <http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.128.6156&rep=rep1&type=pdf>

[15] Agrawal, Nitin; Prabhakaran, Vijayan; Wobber, Ted; Davis, John; Manasse, Mark. Panigrahy, Rina. ''Design Tradeoffs for SSD Performance''. ''Association for Computing Machinery (ACM)'', USENIX 2008 Annual Technical Conference, 2008. <http://portal.acm.org/citation.cfm?id=1404014.1404019>

[16] Lee, Sang-Won, et al. ''A Log Buffer-Based Flash Translation Layer Using Fully-Associative Sector Translation''. ''Association for Computing Machinery (ACM)''. ACM Transactions on Embedded Computing Systems (TECS), Jul 2007. <http://portal.acm.org/citation.cfm?id=1275990>

[17] ''Reach New Heights in Computing Performance''. ''Micron Technology Inc''. Micro Technology Inc, Accessed 14 Oct 2010. <http://www.micron.com/products/solid_state_storage/client_ssd.html>

[18] ''Flash Memories.'' 1 ed. New York: Springer, 1999. Print.

[19] ''Nonvolatile Memory Technologies with Emphasis on Flash: A Comprehensive Guide to Understanding and Using Flash Memory Devices''. ''IEEE Press Series on Microelectronic Systems''. New York: Wiley-Ieee Press, 2008. Print.

[20] ''Nonvolatile Semiconductor Memory Technology: A Comprehensive Guide to Understanding and Using NVSM Devices''. ''IEEE Press Series on Microelectronic Systems''. New York: Wiley-Ieee Press, 1997. Print.

=External links=

Relevant Wikipedia articles: [http://en.wikipedia.org/wiki/Flash_Memory Flash Memory], [http://en.wikipedia.org/wiki/LogFS LogFS], [http://en.wikipedia.org/wiki/Hard_disk Hard Disk Drives], [http://en.wikipedia.org/wiki/Wear_leveling Wear Leveling], [http://en.wikipedia.org/wiki/Hot_spot_%28computer_science%29 Hot Spots], [http://en.wikipedia.org/wiki/Solid-state_drive Sold State Drive].

=old=

Hey guys,

This is what I've got so far... mostly based on wikipedia:

Flash memory has two limitations: it can only be erased in blocks and and it wears out after a certain number of erase cycles. Furthermore, a particular kind of Flash memory (NAND) is not able to provide random access.
As a result of these Flash based file-systems cannot be handled in the same way as disk-based file systems. Here are a few of the key differences:

- Because memory must be erased in blocks, its erasure tends to take up time. Consequently, it is necessary to time the erasures in a way so as not to interfere with the efficiency of the system’s other operations. This is is not a real concern with disk-based file-systems.
- A disk file-system needs to minimize the seeking time, but Flash file-system does not concern itself with this as it doesn’t have a disk.
- A flash system tries to distribute memory in such a way so as not to make a particular block of memory subject to a disproportionally large number of erasures. The purpose of this is to keep the block from wearing out prematurely. The result of it is that memory needs to be distributed differently than in a disk based file-system.
Log-sturctured file systems are thus best suited to dealing with flash memory (they apparently do all of the above things).

For the essay form, I'm thinking of doing a section about traditional hard-disk systems, another about flash-memory and a third about flash systems. At this point, I am imagining the thesis as something like, "Flash systems require a fundamentally different system architecture than disk-based systems due to their need to adapt to the constraints inherent in flash memory: specifically, due to that memory's limited life-span and block-based erasures." The argument would then talk about how these two differences directly lead to a new FS approach.

That's how I see it at the moment. Honestly, I don't like doing research about this kind of stuff, so my data isn't very deep. That said, if you guys could find more info and summarize it, I'm pretty sure that I could synthesize it all into a coherent essay.

Fedor

COMP 3000 Essay 1 2010 Question 10

2010-10-15T03:23:16Z

Gsmith6: /* Answer */

=Question=

How do the constraints of flash storage affect the design of flash-optimized file systems? Explain by contrasting with hard disk-based file systems.

=Answer=
First introduced in the late 80s, Flash-memory is a light, energy-independent, compact, shock-resistant and efficiently readable type of storage. Because of the particular limitations of this kind of memory, flash file systems require a fundamentally different system architecture than disk-based file-systems: these systems need to be designed in light of flash-memory’s limited number of erase-cycles and its need to conduct erasures one entire block at a time. These constraints are a direct result of the same design that gives flash its advantages with regard to [ TO WHAT?] as both are due to [TO WHAT?] . Thus, a typical disk-based file-system is not suitable for working with flash memory as it erases far too frequently and indiscriminately while being simultaneously optimized for other constraints that do not affect flash memory. This means that a different solution is necessary and that solution is the log-based file-system which is far better suited to working with flash memory because it optimizes erasures by [WHAT?].

==Flash Memory==
Flash memory is non-volatile(meaning digital storage that does not require power to retain its memory) storage space that has become more popular recently due to its fast fetch times. There are two basic forms of the flash storage system, NOR and NAND. Each type has its advantages and disadvantages. NOR has the fastest read times, but is much slower at writing. NAND on the other hand has much more capacity, faster write times, is less expensive, and has a much longer life expectancy.[2]

More and more people use flash memory, with many sizes of drives, ranging from a few hundred megabyte USB key, to a few terabyte internal solid-state drive(SSD). Two main reasons for this movement are because of flash's extremely fast read times, and its falling price. A typical flash drive has read speeds of up to 14 times faster than a hard disk drive (HDD).[17]

Although flash drives are exponentially faster than HDDs, they still have not become the main source of data management. The reason for this is because HDDs are simply much cheaper, and flash drives still have many faults. The most critical fault is that each block in flash memory can only be erased approximately 100,000 times.[14] This poses a problem because when modifying a file, even if its a single bit, the entire block must be erased, and rewritten. This erase/rewrite slows down the write operation considerably, making it actually slower to write a file to flash than an HDD.[8]

'''1. What flash storage is, why its good but also why it must have the problems that it does (the assumption is that it must have them, why would it otherwise?) [don't know much about this just now... basics include that there is NOR (reads slightly faster)and NAND (holds more, writes faster, erases much faster, lasts about ten times longer) flash with NAND being especially popular for storage (what's NOR good for?). Here, we'd ideally want to talk about why flash was invented (supposed as an alternative to slow ROM), why it was suitable for that, and how it works on a technical level. Then, we'd want to mention why this technical functionality was innovative and useful but also why it came with two serious set-backs: having a limited-number of re-write cycles and needing to erase a block at a time.]
Either way, Flash storage affords far faster fetch times than the traditional platter-based HDD, and stability of information in a sense. Where the data is not actually stored, but reprogrammed, in a sense, the data is more secure and is less likely to be erased easily. On that note, in order to flip a single bit, that entire block will need to be erased, then reprogrammed. In an 'old' HDD, let's say, When the HDD fails at the end of its life cycle, your data is gone. (unless you're willing to shell out $200/hr to have it recovered, yes I've seen companies in Ottawa that do this.) In a flash HDD, when it reaches the end of its life, it merely becomes read-only. Bugger for Databases, but useful for technical notes and archives, let's say. With today's modern gaming computers, Flash memory can be good on quick load times, however with limited read-writes, it could afford better use to things that are not updated as frequently. I.e... Well I don't have a better example than a webserver hosting a company's CSS and scripts. ~Source: Years in the 'biz
Flash memory started out as a replacement for EPROMs. At the time EPROMs needed a UV photoemission to be erased while flash memory could be erased electronically. The first flash memory product came out in 1988 but it did not take off until the late 1990’s because it could not be reliable produced. NOR and NAND memory is named after the arrangement of the cells in the memory array. NOR based flash memory benefits from having very fast burst read times but slower write times. Due to the structure of NOR memory programs stored in NOR based memory can be executed without being loaded into RAM first. NAND flash memory has a very large storage capacity and can read and write large files relatively fast. NAND is more suited for storage while NOR memory is better suited for direct program execution such as in CMOS chips'''

HDDs use a block system, in which the kernel specifies which blocks to read and write. When using a flash drive, the blocks are emulated and mapped to a physical memory address. It does through what is called a "Transition Layer".

==Traditionally Optimized File Systems==

Since the kernel asks for a block number, a conventional hard disk drive (HDD) file-system is not optimized to work with flash memory. The reason for this is that conventional hard-disks have different constraints from those of flash memory - their primary problem is to reduce seeking time, while the primary problem when working with flash memory is to erase in a minimal and balanced way.

The most consuming process for an HDD is seeking data by relocating the read-head and spinning the magnetic disk. A traditional file system optimizes the way it stores data by placing related blocks close-by on the disk in order to minimize mechanical movement within the HDD. One of the great advantages of flash memory, which accounts for its fast read speed, is that there is no need to seek data physically. This is also why defragmentaion, a procedure used by HDDs to put files into more convenient configurations and thus minimize seeking times, loses its purpose in a flash memory context. Indeed, the unnecessary erasures that it entails are both inefficient and harmful for a flash memory unit.

This comes directly out of flash memory's aforementioned constraints: the slow block-sized erasures and the limited number of erase-cycles. Because of these, a flash optimal file system needs to minimize its erase operations and also to spread out its erasures in such a way as to avoid the formation of hot-spots: sections of memory which have undergone a disproportionately high number of erasures and are thus in danger of burning out. This process of spreading out data is referred to as "wear leveling". To minimize hotspots, a system using flash memory would have to write new data to empty memory blocks. This method would also call for some sort of garbage collection to conduct necessary erasure operations while the system is idle. It makes better sense to do these at this time because of the slow nature of erasures in the flash memory context. Of course, there is no such feature in a traditional HDD file-system.

'''More on this later'''

==Flash Optimized File Systems==

The process of "wear leveling" [WHAT IS THIS?] is achieved through a Log-based File System, often referred to as the Flash Transitional Layer(FTL). Essentially, the drive stores a log that keeps track of how many times each erase sector has been invalidated (or erased). The translational layer has a translation table, where each physical memory address is associated with an emulated block sectors. This allows a traditional file system that uses block sectors to be used on the flash drive. Each block has a flag which keeps track of its state. When a block is being written to, the FTL marks the blocks needed as ''allocated''. This prevents other data being written to the block that has already been allocated. The FTL then goes on to write the data in the allocated blocks. Once it completes the transaction, the system updates the allocated blocks to ''pre-valid''. Once that is completed, the drive marks the invalidated blocks to ''invalid'', while marking the newly written block as ''valid''. This entire flagging process is to ensure that the newly allocated blocks are never mixed up with the invalidated blocks.

===Banks===
The FTL organizes data using structures called banks. When the FTL gets a request to write something to memory, it uses a bank list to determine which area of the drive should be used. Essentially a bank is a group of sequential addresses, that keeps track of when it was last updated using timestamps. The FTL will only write to that bank, and once there is not enough space to write anymore, it switches out the current bank for the one with the most available space. When cleaning up the bank, the system puts it into what is called the Cleaning Bank List and removes it from the Bank List, thus avoiding any chance of some data being written to that bank while something is being erased.

===Cleaner===
When the FTL realizes that there is not enough room to write new data onto the drive, it runs a garbage collection routine. This routine selects a segment to be cleaned, copies all of the valid data into a new segment, then erases everything in the old segment. This frees up the otherwise useless invalidated blocks and by not erasing every block as soon as it becomes invalidated, it saves on the amount of times that the expensive erase operation is called. The kernel can preemptively clean the drive when the system is idle.

=Conclusion=

In this way, thanks to its [WHATEVER MAKES LOG FSs ACTUALLY GOOD AT DEALING WITH FLASH], the log-based file-system is far better suited to working with flash memory than a traditional HDD file system. The latter is utterly unfit for this task due to its placing primacy on the minimization of seeks rather than on the minimization and management of erasures. Dealing smartly with erasures is extremely important for a flash memory file system, as that memory type's particular weaknesses, the limited number of erasure cycles, the necessity to erase by the block and the relative slowness of the erasures themselves, all have to do with erasing. A good flash memory file system must therefore be built with the aim of making the best of these weaknesses and this is precisely the reason why older disk-based file systems are not suitable for flash memory while log-based file systems are. [INSPIRATIONAL LAST WORDS]

=Questions=

=References=

[1] Kim, Han-joon; Lee, Sang-goo. ''A New Flash Memory Management for Flash Storage System''. ''IEEExplore''. Dept. of Comput. Sci., Seoul Nat. Univ., 06 Aug 2002. <http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=812717&tag=1#>

[2] Smith, Lance. ''NAND Flash Solid State Storage Performance and Capability''. ''Flash Memory Summit''. SNIA Education Committee, 18 Aug 2009. <http://www.flashmemorysummit.com/English/Collaterals/Proceedings/2009/20090812_T1B_Smith.pdf>

[3] Chang, LiPin. ''On Efficient Wear Leveling for Large-Scale Flash-Memory Storage Systems''. ''Association for Computing Machinery (ACM)''. Dept. of Comput. Sci.,Nat. ChiaoTung Univ., 15 Mar 2007. <http://portal.acm.org/citation.cfm?id=1244248>

[4] Nath, Suman; Gibbons, Phillip. ''Online maintenance of very large random samples on flash storage''. ''Association for Computing Machinery (ACM)''. The VLDB Journal, 27 Jul 2007. <http://portal.acm.org/citation.cfm?id=1731355>

[5] Lim, Seung-Ho; Park; Kyu-Ho. ''An Efficient NAND Flash File System for Flash Memory Storage''. ''CORE Laboratory''. IEEE Transactions On Computers, Jul 2006. <http://vlsi.kaist.ac.kr/paper_list/2006_TC_CFFS.pdf>

[6] ''NAND vs. NOR Flash Memory Technology Overview''. ''RMG and Associates''. Toshiba America, accessed 14 Oct 2010. <http://maltiel-consulting.com/NAND_vs_NOR_Flash_Memory_Technology_Overview_Read_Write_Erase_speed_for_SLC_MLC_semiconductor_consulting_expert.pdf>

[7] Bez, Roberto; Camerlenghi, Emilio; Modelli, Alberto; Visconti, Angelo. ''Introduction to Flash Memory''. ''IEEExplore''. STMicroelectronics, 21 May 2003. <http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1199079&tag=1>

[8] Kawaguchi, Atsuo; Nishioka, Shingo; Motoda Hiroshi. ''A Flash-Memory Based File System''. ''CiteSeerX'' Advanced Research laboratory, Hitachi, Ltd., 1995. <http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.160.5142>

[9] Rosenblum, Mendel; Ouserhout, John. ''The Design and Implementation of a Log-structured File System''. ''Association for Computing Machinery (ACM)''. University of California at Berkeley, Feb 1992. <http://portal.acm.org/citation.cfm?id=146943&coll=GUIDE&dl=GUIDE&CFID=108397378&CFTOKEN=72657973&ret=1#Fulltext>

[10] Shimpi, Anand. ''Intel X25-M SSD: Intel Delivers One of the World's Fastest Drives''. ''AnAndTech''. AnAndTech, 8 Sep 2008. <http://www.anandtech.com/show/2614>

[11] Shimpi, Anand. ''The SSD Relapse: Understanding and Choosing the Best SSD''. ''AnAndTech''. AnAndTech, 30 Aug 2009. <http://www.anandtech.com/show/2829>

[12] Shimpi, Anand. ''The SSD Anthology: Understanding SSDs and New Drives from OCZ''. ''AnAndTech''. AnAndTech, 18 Mar 2009. <http://www.anandtech.com/show/2738>

[13] Corbet, Jonathan. ''Solid-State Storage Devices and the Block Layer''. ''Linux Weekly News''. Linux Weekly News, 4 Oct 2010. <http://lwn.net/Articles/408428/>

[14] Woodhouse, David. ''JFFS : The Journalling Flash File System''. ''CiteSeerX''. Red Hat, Inc, Accessed 14 Oct 2010. <http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.128.6156&rep=rep1&type=pdf>

[15] Agrawal, Nitin; Prabhakaran, Vijayan; Wobber, Ted; Davis, John; Manasse, Mark. Panigrahy, Rina. ''Design Tradeoffs for SSD Performance''. ''Association for Computing Machinery (ACM)'', USENIX 2008 Annual Technical Conference, 2008. <http://portal.acm.org/citation.cfm?id=1404014.1404019>

[16] Lee, Sang-Won, et al. ''A Log Buffer-Based Flash Translation Layer Using Fully-Associative Sector Translation''. ''Association for Computing Machinery (ACM)''. ACM Transactions on Embedded Computing Systems (TECS), Jul 2007. <http://portal.acm.org/citation.cfm?id=1275990>

[17] ''Reach New Heights in Computing Performance''. ''Micron Technology Inc''. Micro Technology Inc, Accessed 14 Oct 2010. <http://www.micron.com/products/solid_state_storage/client_ssd.html>

[18] ''Flash Memories.'' 1 ed. New York: Springer, 1999. Print.

[19] ''Nonvolatile Memory Technologies with Emphasis on Flash: A Comprehensive Guide to Understanding and Using Flash Memory Devices''. ''IEEE Press Series on Microelectronic Systems''. New York: Wiley-Ieee Press, 2008. Print.

[20] ''Nonvolatile Semiconductor Memory Technology: A Comprehensive Guide to Understanding and Using NVSM Devices''. ''IEEE Press Series on Microelectronic Systems''. New York: Wiley-Ieee Press, 1997. Print.

=External links=

Relevant Wikipedia articles: [http://en.wikipedia.org/wiki/Flash_Memory Flash Memory], [http://en.wikipedia.org/wiki/LogFS LogFS], [http://en.wikipedia.org/wiki/Hard_disk Hard Disk Drives], [http://en.wikipedia.org/wiki/Wear_leveling Wear Leveling], [http://en.wikipedia.org/wiki/Hot_spot_%28computer_science%29 Hot Spots], [http://en.wikipedia.org/wiki/Solid-state_drive Sold State Drive].

=old=

Hey guys,

This is what I've got so far... mostly based on wikipedia:

Flash memory has two limitations: it can only be erased in blocks and and it wears out after a certain number of erase cycles. Furthermore, a particular kind of Flash memory (NAND) is not able to provide random access.
As a result of these Flash based file-systems cannot be handled in the same way as disk-based file systems. Here are a few of the key differences:

- Because memory must be erased in blocks, its erasure tends to take up time. Consequently, it is necessary to time the erasures in a way so as not to interfere with the efficiency of the system’s other operations. This is is not a real concern with disk-based file-systems.
- A disk file-system needs to minimize the seeking time, but Flash file-system does not concern itself with this as it doesn’t have a disk.
- A flash system tries to distribute memory in such a way so as not to make a particular block of memory subject to a disproportionally large number of erasures. The purpose of this is to keep the block from wearing out prematurely. The result of it is that memory needs to be distributed differently than in a disk based file-system.
Log-sturctured file systems are thus best suited to dealing with flash memory (they apparently do all of the above things).

For the essay form, I'm thinking of doing a section about traditional hard-disk systems, another about flash-memory and a third about flash systems. At this point, I am imagining the thesis as something like, "Flash systems require a fundamentally different system architecture than disk-based systems due to their need to adapt to the constraints inherent in flash memory: specifically, due to that memory's limited life-span and block-based erasures." The argument would then talk about how these two differences directly lead to a new FS approach.

That's how I see it at the moment. Honestly, I don't like doing research about this kind of stuff, so my data isn't very deep. That said, if you guys could find more info and summarize it, I'm pretty sure that I could synthesize it all into a coherent essay.

Fedor

COMP 3000 Essay 1 2010 Question 10

2010-10-15T03:00:11Z

Gsmith6: /* Flash Optimized File Systems */

=Question=

How do the constraints of flash storage affect the design of flash-optimized file systems? Explain by contrasting with hard disk-based file systems.

=Answer=
First introduced in the late 80s, Flash-memory is a light, energy-independent, compact, shock-resistant and efficiently readable type of storage. Because of the particular limitations of this kind of memory, flash file systems require a fundamentally different system architecture than disk-based file-systems: these systems need to be designed in light of flash-memory’s limited number of erase-cycles and its need to conduct erasures one entire block at a time. These constraints are a direct result of the same design that gives flash its advantages with regard to [ TO WHAT?] as both are due to [TO WHAT?] . Thus, a typical disk-based file-system is not suitable for working with flash memory as it erases far too frequently and indiscriminately while being simultaneously optimized for other constraints that do not affect flash memory. This means that a different solution is necessary and that solution is the log-based file-system which is far better suited to working with flash memory because it optimizes erasures by [WHAT?].

==Flash Memory==
Flash memory is non-volatile(meaning digital storage that does not require power to retain its memory) storage space that has become more popular recently due to its fast fetch times. There are two basic forms of the flash storage system, NOR and NAND. Each type has its advantages and disadvantages. NOR has the fastest read times, but is much slower at writing. NAND on the other hand has much more capacity, faster write times, is less expensive, and has a much longer life expectancy.[2]

More and more people use flash memory, with many sizes of drives, ranging from a few hundred megabyte USB key, to a few terabyte internal solid-state drive(SSD). Two main reasons for this movement are because of flash's extremely fast read times, and its falling price. A typical flash drive has read speeds of up to 14 times faster than a hard disk drive (HDD).[17]

Although flash drives are exponentially faster than HDDs, they still have not become the main source of data management. The reason for this is because HDDs are simply much cheaper, and flash drives still have many faults. The most critical fault is that each block in flash memory can only be erased approximately 100,000 times.[14] This poses a problem because when modifying a file, even if its a single bit, the entire block must be erased, and rewritten. This erase/rewrite slows down the write operation considerably, making it actually slower to write a file to flash than an HDD.[8]

'''Possibly more on this later'''

HDDs use a block system, in which the kernel specifies which blocks to read and write. When using a flash drive, the blocks are emulated and mapped to a physical memory address. It does through what is called a "Transition Layer".

==Traditionally Optimized File Systems==

Since the kernel asks for a block number, a conventional hard disk drive (HDD) file-system is not optimized to work with flash memory. The reason for this is that conventional hard-disks have different constraints from those of flash memory - their primary problem is to reduce seeking time, while the primary problem when working with flash memory is to erase in a minimal and balanced way.

The most consuming process for an HDD is seeking data by relocating the read-head and spinning the magnetic disk. A traditional file system optimizes the way it stores data by placing related blocks close-by on the disk in order to minimize mechanical movement within the HDD. One of the great advantages of flash memory, which accounts for its fast read speed, is that there is no need to seek data physically. This is also why defragmentaion, a procedure used by HDDs to put files into more convenient configurations and thus minimize seeking times, loses its purpose in a flash memory context. Indeed, the unnecessary erasures that it entails are both inefficient and harmful for a flash memory unit.

This comes directly out of flash memory's aforementioned constraints: the slow block-sized erasures and the limited number of erase-cycles. Because of these, a flash optimal file system needs to minimize its erase operations and also to spread out its erasures in such a way as to avoid the formation of hot-spots: sections of memory which have undergone a disproportionately high number of erasures and are thus in danger of burning out. This process of spreading out data is referred to as "wear leveling". To minimize hotspots, a system using flash memory would have to write new data to empty memory blocks. This method would also call for some sort of garbage collection to conduct necessary erasure operations while the system is idle. It makes better sense to do these at this time because of the slow nature of erasures in the flash memory context. Of course, there is no such feature in a traditional HDD file-system.

'''More on this later'''

==Flash Optimized File Systems==

The process of "wear leveling" [WHAT IS THIS?] is achieved through a Log-based File System, often referred to as the Flash Transitional Layer(FTL). Essentially, the drive stores a log that keeps track of how many times each erase sector has been invalidated (or erased). The translational layer has a translation table, where each physical memory address is associated with an emulated block sectors. This allows a traditional file system that uses block sectors to be used on the flash drive. Each block has a flag which keeps track of its state. When a block is being written to, the FTL marks the blocks needed as ''allocated''. This prevents other data being written to the block that has already been allocated. The FTL then goes on to write the data in the allocated blocks. Once it completes the transaction, the system updates the allocated blocks to ''pre-valid''. Once that is completed, the drive marks the invalidated blocks to ''invalid'', while marking the newly written block as ''valid''. This entire flagging process is to ensure that the newly allocated blocks are never mixed up with the invalidated blocks.

===Banks===
The FTL organizes data using structures called banks. When the FTL gets a request to write something to memory, it uses a bank list to determine which area of the drive should be used. Essentially a bank is a group of sequential addresses, that keeps track of when it was last updated using timestamps. The FTL will only write to that bank, and once there is not enough space to write anymore, it switches out the current bank for the one with the most available space. When cleaning up the bank, the system puts it into what is called the Cleaning Bank List and removes it from the Bank List, thus avoiding any chance of some data being written to that bank while something is being erased.

===Cleaner===
When the FTL realizes that there is not enough room to write new data onto the drive, it runs a garbage collection routine. This routine selects a segment to be cleaned, copies all of the valid data into a new segment, then erases everything in the old segment. This frees up the otherwise useless invalidated blocks and by not erasing every block as soon as it becomes invalidated, it saves on the amount of times that the expensive erase operation is called. The kernel can preemptively clean the drive when the system is idle.

=Conclusion=

=Questions=

=References=

[1] Kim, Han-joon; Lee, Sang-goo. ''A New Flash Memory Management for Flash Storage System''. ''IEEExplore''. Dept. of Comput. Sci., Seoul Nat. Univ., 06 Aug 2002. <http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=812717&tag=1#>

[2] Smith, Lance. ''NAND Flash Solid State Storage Performance and Capability''. ''Flash Memory Summit''. SNIA Education Committee, 18 Aug 2009. <http://www.flashmemorysummit.com/English/Collaterals/Proceedings/2009/20090812_T1B_Smith.pdf>

[3] Chang, LiPin. ''On Efficient Wear Leveling for Large-Scale Flash-Memory Storage Systems''. ''Association for Computing Machinery (ACM)''. Dept. of Comput. Sci.,Nat. ChiaoTung Univ., 15 Mar 2007. <http://portal.acm.org/citation.cfm?id=1244248>

[4] Nath, Suman; Gibbons, Phillip. ''Online maintenance of very large random samples on flash storage''. ''Association for Computing Machinery (ACM)''. The VLDB Journal, 27 Jul 2007. <http://portal.acm.org/citation.cfm?id=1731355>

[5] Lim, Seung-Ho; Park; Kyu-Ho. ''An Efficient NAND Flash File System for Flash Memory Storage''. ''CORE Laboratory''. IEEE Transactions On Computers, Jul 2006. <http://vlsi.kaist.ac.kr/paper_list/2006_TC_CFFS.pdf>

[6] ''NAND vs. NOR Flash Memory Technology Overview''. ''RMG and Associates''. Toshiba America, accessed 14 Oct 2010. <http://maltiel-consulting.com/NAND_vs_NOR_Flash_Memory_Technology_Overview_Read_Write_Erase_speed_for_SLC_MLC_semiconductor_consulting_expert.pdf>

[7] Bez, Roberto; Camerlenghi, Emilio; Modelli, Alberto; Visconti, Angelo. ''Introduction to Flash Memory''. ''IEEExplore''. STMicroelectronics, 21 May 2003. <http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1199079&tag=1>

[8] Kawaguchi, Atsuo; Nishioka, Shingo; Motoda Hiroshi. ''A Flash-Memory Based File System''. ''CiteSeerX'' Advanced Research laboratory, Hitachi, Ltd., 1995. <http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.160.5142>

[9] Rosenblum, Mendel; Ouserhout, John. ''The Design and Implementation of a Log-structured File System''. ''Association for Computing Machinery (ACM)''. University of California at Berkeley, Feb 1992. <http://portal.acm.org/citation.cfm?id=146943&coll=GUIDE&dl=GUIDE&CFID=108397378&CFTOKEN=72657973&ret=1#Fulltext>

[10] Shimpi, Anand. ''Intel X25-M SSD: Intel Delivers One of the World's Fastest Drives''. ''AnAndTech''. AnAndTech, 8 Sep 2008. <http://www.anandtech.com/show/2614>

[11] Shimpi, Anand. ''The SSD Relapse: Understanding and Choosing the Best SSD''. ''AnAndTech''. AnAndTech, 30 Aug 2009. <http://www.anandtech.com/show/2829>

[12] Shimpi, Anand. ''The SSD Anthology: Understanding SSDs and New Drives from OCZ''. ''AnAndTech''. AnAndTech, 18 Mar 2009. <http://www.anandtech.com/show/2738>

[13] Corbet, Jonathan. ''Solid-State Storage Devices and the Block Layer''. ''Linux Weekly News''. Linux Weekly News, 4 Oct 2010. <http://lwn.net/Articles/408428/>

[14] Woodhouse, David. ''JFFS : The Journalling Flash File System''. ''CiteSeerX''. Red Hat, Inc, Accessed 14 Oct 2010. <http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.128.6156&rep=rep1&type=pdf>

[15] Agrawal, Nitin; Prabhakaran, Vijayan; Wobber, Ted; Davis, John; Manasse, Mark. Panigrahy, Rina. ''Design Tradeoffs for SSD Performance''. ''Association for Computing Machinery (ACM)'', USENIX 2008 Annual Technical Conference, 2008. <http://portal.acm.org/citation.cfm?id=1404014.1404019>

[16] Lee, Sang-Won, et al. ''A Log Buffer-Based Flash Translation Layer Using Fully-Associative Sector Translation''. ''Association for Computing Machinery (ACM)''. ACM Transactions on Embedded Computing Systems (TECS), Jul 2007. <http://portal.acm.org/citation.cfm?id=1275990>

[17] ''Reach New Heights in Computing Performance''. ''Micron Technology Inc''. Micro Technology Inc, Accessed 14 Oct 2010. <http://www.micron.com/products/solid_state_storage/client_ssd.html>

[18] ''Flash Memories.'' 1 ed. New York: Springer, 1999. Print.

[19] ''Nonvolatile Memory Technologies with Emphasis on Flash: A Comprehensive Guide to Understanding and Using Flash Memory Devices''. ''IEEE Press Series on Microelectronic Systems''. New York: Wiley-Ieee Press, 2008. Print.

[20] ''Nonvolatile Semiconductor Memory Technology: A Comprehensive Guide to Understanding and Using NVSM Devices''. ''IEEE Press Series on Microelectronic Systems''. New York: Wiley-Ieee Press, 1997. Print.

=External links=

Relevant Wikipedia articles: [http://en.wikipedia.org/wiki/Flash_Memory Flash Memory], [http://en.wikipedia.org/wiki/LogFS LogFS], [http://en.wikipedia.org/wiki/Hard_disk Hard Disk Drives], [http://en.wikipedia.org/wiki/Wear_leveling Wear Leveling], [http://en.wikipedia.org/wiki/Hot_spot_%28computer_science%29 Hot Spots], [http://en.wikipedia.org/wiki/Solid-state_drive Sold State Drive].

=old=

Hey guys,

This is what I've got so far... mostly based on wikipedia:

Flash memory has two limitations: it can only be erased in blocks and and it wears out after a certain number of erase cycles. Furthermore, a particular kind of Flash memory (NAND) is not able to provide random access.
As a result of these Flash based file-systems cannot be handled in the same way as disk-based file systems. Here are a few of the key differences:

- Because memory must be erased in blocks, its erasure tends to take up time. Consequently, it is necessary to time the erasures in a way so as not to interfere with the efficiency of the system’s other operations. This is is not a real concern with disk-based file-systems.
- A disk file-system needs to minimize the seeking time, but Flash file-system does not concern itself with this as it doesn’t have a disk.
- A flash system tries to distribute memory in such a way so as not to make a particular block of memory subject to a disproportionally large number of erasures. The purpose of this is to keep the block from wearing out prematurely. The result of it is that memory needs to be distributed differently than in a disk based file-system.
Log-sturctured file systems are thus best suited to dealing with flash memory (they apparently do all of the above things).

For the essay form, I'm thinking of doing a section about traditional hard-disk systems, another about flash-memory and a third about flash systems. At this point, I am imagining the thesis as something like, "Flash systems require a fundamentally different system architecture than disk-based systems due to their need to adapt to the constraints inherent in flash memory: specifically, due to that memory's limited life-span and block-based erasures." The argument would then talk about how these two differences directly lead to a new FS approach.

That's how I see it at the moment. Honestly, I don't like doing research about this kind of stuff, so my data isn't very deep. That said, if you guys could find more info and summarize it, I'm pretty sure that I could synthesize it all into a coherent essay.

Fedor

COMP 3000 Essay 1 2010 Question 10

2010-10-15T02:39:36Z

Gsmith6: /* Flash Optimized File Systems */

=Question=

How do the constraints of flash storage affect the design of flash-optimized file systems? Explain by contrasting with hard disk-based file systems.

=Answer=
First introduced in the late 80s, Flash-memory is a light, energy-independent, compact, shock-resistant and efficiently readable type of storage. Because of the particular limitations of this kind of memory, flash file systems require a fundamentally different system architecture than disk-based file-systems: these systems need to be designed in light of flash-memory’s limited number of erase-cycles and its need to conduct erasures one entire block at a time. These constraints are a direct result of the same design that gives flash its advantages with regard to [ TO WHAT?] as both are due to [TO WHAT?] . Thus, a typical disk-based file-system is not suitable for working with flash memory as it erases far too frequently and indiscriminately while being simultaneously optimized for other constraints that do not affect flash memory. This means that a different solution is necessary and that solution is the log-based file-system which is far better suited to working with flash memory because it optimizes erasures by [WHAT?].

==Flash Memory==
Flash memory is non-volatile(meaning digital storage that does not require power to retain its memory) storage space that has become more popular recently due to its fast fetch times. There are two basic forms of the flash storage system, NOR and NAND. Each type has its advantages and disadvantages. NOR has the fastest read times, but is much slower at writing. NAND on the other hand has much more capacity, faster write times, is less expensive, and has a much longer life expectancy.[2]

More and more people use flash memory, with many sizes of drives, ranging from a few hundred megabyte USB key, to a few terabyte internal solid-state drive(SSD). Two main reasons for this movement are because of flash's extremely fast read times, and its falling price. A typical flash drive has read speeds of up to 14 times faster than a hard disk drive (HDD).[17]

Although flash drives are exponentially faster than HDDs, they still have not become the main source of data management. The reason for this is because HDDs are simply much cheaper, and flash drives still have many faults. The most critical fault is that each block in flash memory can only be erased approximately 100,000 times.[14] This poses a problem because when modifying a file, even if its a single bit, the entire block must be erased, and rewritten. This erase/rewrite slows down the write operation considerably, making it actually slower to write a file to flash than an HDD.[8]

'''Possibly more on this later'''

HDDs use a block system, in which the kernel specifies which blocks to read and write. When using a flash drive, the blocks are emulated and mapped to a physical memory address. It does through what is called a "Transition Layer".

==Traditionally Optimized File Systems==

Since the kernel asks for a block number, a conventional hard disk drive (HDD) file-system is not optimized to work with flash memory. The reason for this is that conventional hard-disks have different constraints from those of flash memory - their primary problem is to reduce seeking time, while the primary problem when working with flash memory is to erase in a minimal and balanced way.

The most consuming process for an HDD is seeking data by relocating the read-head and spinning the magnetic disk. A traditional file system optimizes the way it stores data by placing related blocks close-by on the disk in order to minimize mechanical movement within the HDD. One of the great advantages of flash memory, which accounts for its fast read speed, is that there is no need to seek data physically. This is also why defragmentaion, a procedure used by HDDs to put files into more convenient configurations and thus minimize seeking times, loses its purpose in a flash memory context. Indeed, the unnecessary erasures that it entails are both inefficient and harmful for a flash memory unit.

This comes directly out of flash memory's aforementioned constraints: the slow block-sized erasures and the limited number of erase-cycles. Because of these, a flash optimal file system needs to minimize its erase operations and also to spread out its erasures in such a way as to avoid the formation of hot-spots: sections of memory which have undergone a disproportionately high number of erasures and are thus in danger of burning out. This process of spreading out data is referred to as "wear leveling". To minimize hotspots, a system using flash memory would have to write new data to empty memory blocks. This method would also call for some sort of garbage collection to conduct necessary erasure operations while the system is idle. It makes better sense to do these at this time because of the slow nature of erasures in the flash memory context. Of course, there is no such feature in a traditional HDD file-system.

'''More on this later'''

==Flash Optimized File Systems==

The process of "wear leveling" is achieved through a Log-based File System, often referred to as the Flash Transitional Layer(FTL). Essentially, the drive stores a log that keeps track of how many times each erase sector has been invalidated (or erased). The translational layer has a translation table, where each physical memory address is associated with an emulated block sectors. This allows a traditional file system that uses block sectors to be used on the flash drive.

===Banks===
The FTL organizes data using structures called banks. When the FTL gets a request to write something to memory, it uses a bank list to determine which area of the drive should be used. Essentially a bank is a group of sequential addresses, that keeps track of when it was last updated using timestamps. The FTL will only write to that bank, and once there is not enough space to write anymore, it switches out the current bank for the one with the most available space.

=Questions=

=References=

[1] Kim, Han-joon; Lee, Sang-goo. ''A New Flash Memory Management for Flash Storage System''. ''IEEExplore''. Dept. of Comput. Sci., Seoul Nat. Univ., 06 Aug 2002. <http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=812717&tag=1#>

[2] Smith, Lance. ''NAND Flash Solid State Storage Performance and Capability''. ''Flash Memory Summit''. SNIA Education Committee, 18 Aug 2009. <http://www.flashmemorysummit.com/English/Collaterals/Proceedings/2009/20090812_T1B_Smith.pdf>

[3] Chang, LiPin. ''On Efficient Wear Leveling for Large-Scale Flash-Memory Storage Systems''. ''Association for Computing Machinery (ACM)''. Dept. of Comput. Sci.,Nat. ChiaoTung Univ., 15 Mar 2007. <http://portal.acm.org/citation.cfm?id=1244248>

[4] Nath, Suman; Gibbons, Phillip. ''Online maintenance of very large random samples on flash storage''. ''Association for Computing Machinery (ACM)''. The VLDB Journal, 27 Jul 2007. <http://portal.acm.org/citation.cfm?id=1731355>

[5] Lim, Seung-Ho; Park; Kyu-Ho. ''An Efficient NAND Flash File System for Flash Memory Storage''. ''CORE Laboratory''. IEEE Transactions On Computers, Jul 2006. <http://vlsi.kaist.ac.kr/paper_list/2006_TC_CFFS.pdf>

[6] ''NAND vs. NOR Flash Memory Technology Overview''. ''RMG and Associates''. Toshiba America, accessed 14 Oct 2010. <http://maltiel-consulting.com/NAND_vs_NOR_Flash_Memory_Technology_Overview_Read_Write_Erase_speed_for_SLC_MLC_semiconductor_consulting_expert.pdf>

[7] Bez, Roberto; Camerlenghi, Emilio; Modelli, Alberto; Visconti, Angelo. ''Introduction to Flash Memory''. ''IEEExplore''. STMicroelectronics, 21 May 2003. <http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1199079&tag=1>

[8] Kawaguchi, Atsuo; Nishioka, Shingo; Motoda Hiroshi. ''A Flash-Memory Based File System''. ''CiteSeerX'' Advanced Research laboratory, Hitachi, Ltd., 1995. <http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.160.5142>

[9] Rosenblum, Mendel; Ouserhout, John. ''The Design and Implementation of a Log-structured File System''. ''Association for Computing Machinery (ACM)''. University of California at Berkeley, Feb 1992. <http://portal.acm.org/citation.cfm?id=146943&coll=GUIDE&dl=GUIDE&CFID=108397378&CFTOKEN=72657973&ret=1#Fulltext>

[10] Shimpi, Anand. ''Intel X25-M SSD: Intel Delivers One of the World's Fastest Drives''. ''AnAndTech''. AnAndTech, 8 Sep 2008. <http://www.anandtech.com/show/2614>

[11] Shimpi, Anand. ''The SSD Relapse: Understanding and Choosing the Best SSD''. ''AnAndTech''. AnAndTech, 30 Aug 2009. <http://www.anandtech.com/show/2829>

[12] Shimpi, Anand. ''The SSD Anthology: Understanding SSDs and New Drives from OCZ''. ''AnAndTech''. AnAndTech, 18 Mar 2009. <http://www.anandtech.com/show/2738>

[13] Corbet, Jonathan. ''Solid-State Storage Devices and the Block Layer''. ''Linux Weekly News''. Linux Weekly News, 4 Oct 2010. <http://lwn.net/Articles/408428/>

[14] Woodhouse, David. ''JFFS : The Journalling Flash File System''. ''CiteSeerX''. Red Hat, Inc, Accessed 14 Oct 2010. <http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.128.6156&rep=rep1&type=pdf>

[15] Agrawal, Nitin; Prabhakaran, Vijayan; Wobber, Ted; Davis, John; Manasse, Mark. Panigrahy, Rina. ''Design Tradeoffs for SSD Performance''. ''Association for Computing Machinery (ACM)'', USENIX 2008 Annual Technical Conference, 2008. <http://portal.acm.org/citation.cfm?id=1404014.1404019>

[16] Lee, Sang-Won, et al. ''A Log Buffer-Based Flash Translation Layer Using Fully-Associative Sector Translation''. ''Association for Computing Machinery (ACM)''. ACM Transactions on Embedded Computing Systems (TECS), Jul 2007. <http://portal.acm.org/citation.cfm?id=1275990>

[17] ''Reach New Heights in Computing Performance''. ''Micron Technology Inc''. Micro Technology Inc, Accessed 14 Oct 2010. <http://www.micron.com/products/solid_state_storage/client_ssd.html>

[18] ''Flash Memories.'' 1 ed. New York: Springer, 1999. Print.

[19] ''Nonvolatile Memory Technologies with Emphasis on Flash: A Comprehensive Guide to Understanding and Using Flash Memory Devices''. ''IEEE Press Series on Microelectronic Systems''. New York: Wiley-Ieee Press, 2008. Print.

[20] ''Nonvolatile Semiconductor Memory Technology: A Comprehensive Guide to Understanding and Using NVSM Devices''. ''IEEE Press Series on Microelectronic Systems''. New York: Wiley-Ieee Press, 1997. Print.

=External links=

Relevant Wikipedia articles: [http://en.wikipedia.org/wiki/Flash_Memory Flash Memory], [http://en.wikipedia.org/wiki/LogFS LogFS], [http://en.wikipedia.org/wiki/Hard_disk Hard Disk Drives], [http://en.wikipedia.org/wiki/Wear_leveling Wear Leveling], [http://en.wikipedia.org/wiki/Hot_spot_%28computer_science%29 Hot Spots], [http://en.wikipedia.org/wiki/Solid-state_drive Sold State Drive].

=old=

Hey guys,

This is what I've got so far... mostly based on wikipedia:

Flash memory has two limitations: it can only be erased in blocks and and it wears out after a certain number of erase cycles. Furthermore, a particular kind of Flash memory (NAND) is not able to provide random access.
As a result of these Flash based file-systems cannot be handled in the same way as disk-based file systems. Here are a few of the key differences:

- Because memory must be erased in blocks, its erasure tends to take up time. Consequently, it is necessary to time the erasures in a way so as not to interfere with the efficiency of the system’s other operations. This is is not a real concern with disk-based file-systems.
- A disk file-system needs to minimize the seeking time, but Flash file-system does not concern itself with this as it doesn’t have a disk.
- A flash system tries to distribute memory in such a way so as not to make a particular block of memory subject to a disproportionally large number of erasures. The purpose of this is to keep the block from wearing out prematurely. The result of it is that memory needs to be distributed differently than in a disk based file-system.
Log-sturctured file systems are thus best suited to dealing with flash memory (they apparently do all of the above things).

For the essay form, I'm thinking of doing a section about traditional hard-disk systems, another about flash-memory and a third about flash systems. At this point, I am imagining the thesis as something like, "Flash systems require a fundamentally different system architecture than disk-based systems due to their need to adapt to the constraints inherent in flash memory: specifically, due to that memory's limited life-span and block-based erasures." The argument would then talk about how these two differences directly lead to a new FS approach.

That's how I see it at the moment. Honestly, I don't like doing research about this kind of stuff, so my data isn't very deep. That said, if you guys could find more info and summarize it, I'm pretty sure that I could synthesize it all into a coherent essay.

Fedor

Talk:COMP 3000 Essay 1 2010 Question 10

2010-10-15T02:39:01Z

Gsmith6:

COMP 3000 Essay 1 2010 Question 10

2010-10-15T02:10:16Z

Gsmith6: /* Flash Optimized File Systems */

=Question=

How do the constraints of flash storage affect the design of flash-optimized file systems? Explain by contrasting with hard disk-based file systems.

=Answer=
First introduced in the late 80s, Flash-memory is a light, energy-independent, compact, shock-resistent and efficiently readable type of storage. Because of the particular limitations of this kind of memory, flash file systems require a fundamentally different system architecture than disk-based file-systems: these systems need to be designed in light of flash-memory’s limited number of erase-cycles and its need to conduct erasures one entire block at a time. These constraints are a direct result of the same design that gives flash its advantages with regard to [ TO WHAT?] as both are due to [TO WHAT?] . Thus, a typical disk-based file-system is not suitable for working with flash memory as it erases far too frequently and indiscriminently while being simultaneously optimized for other constraints that do not affect flash memory. This means that a different solution is necessary and that solution is the log-based file-system which is far better suited to working with flash memory because it optimizes erasures by [WHAT?].

==Flash Memory==
Flash memory is non-volatile(meaning digital storage that does not require power to retain its memory) storage space that has become more popular recently due to its fast fetch times. There are two basic forms of the flash storage system, NOR and NAND. Each type has its advantages and disadvantages. NOR has the fastest read times, but is much slower at writing. NAND on the other hand has much more capacity, faster write times, is less expensive, and has a much longer life expectancy.[2]

More and more people use flash memory, with many sizes of drives, ranging from a few hundred megabyte USB key, to a few terrabyte internal solid-state drive(SSD). Two main reasons for this movement are because of flash's extremely fast read times, and its falling price. A typical flash drive has read speeds of up to 14 times faster than a hard disk drive (HDD).[17]

Although flash drives are exponentially faster than HDDs, they still have not become the main source of data management. The reason for this is because HDDs are simply much cheaper, and flash drives still have many faults. The most critical fault is that each block in flash memory can only be erased approximately 100,000 times.[14] This poses a problem because when modifying a file, even if its a single bit, the entire block must be erased, and rewritten. This erase/rewrite slows down the write operation considerably, making it actually slower to write a file to flash than an HDD.[8]

'''Possibly more on this later'''

HDDs use a block system, in which the kernel specifies which blocks to read and write. When using a flash drive, the blocks are emulated and mapped to a physical memory address. It does through what is called a "Transition Layer".

==Traditionally Optimized File Systems==

Since the kernel asks for a block number, a conventional hard disk drive (HDD) file-system is not optimized to work with flash memory. The reason for this is that conventional hard-disks have different constraints from those of flash memory - their primary problem is to reduce seeking time, while the primary problem when working with flash memory is to erase in a minimal and balanced way.

The most consuming process for an HDD is seeking data by relocating the read-head and spinning the magnetic disk. A traditional file system optimizes the way it stores data by placing related blocks close-by on the disk in order to minimize mechanical movement within the HDD. One of the great advantages of flash memory, which accounts for its fast read speed, is that there is no need to seek data physically. This is also why defragmentaion, a procedure used by HDDs to put files into more convenient configurations and thus minimize seeking times, loses its purpose in a flash memory context. Indeed, the unnecessary erasures that it entails are both inefficient and harmful for a flash memory unit.

This comes direclty out of flash memory's aforementioned constraints: the slow block-sized erasures and the limited number of erase-cycles. Because of these, a flash optimal file system needs to minimize its erase operations and also to spread out its erasures in such a way as to avoid the formation of hot-spots: sections of memory which have undergone a disproportionately high number of erasures and are thus in danger of burning out. This process of spreading out data is referred to as "wear leveling". To minimize hotspots, a system using flash memory would have to write new data to empty memory blocks. This method would also call for some sort of garbage collection to conduct necessary erasure operations while the system is idle. It makes better sense to do these at this time because of the slow nature of erasures in the flash memory context. Of course, there is no such feature in a traditional HDD file-system.

'''More on this later'''

==Flash Optimized File Systems==

The process of "wear leveling" is achieved through a Log-based File System, often referred to as the Flash Transitional Layer(FTL). Essentially, the drive stores a log that keeps track of how many times each erase sector has been invalidated (or erased). The translational layer has a translation table, where each physical memory address is associated with an emulated block sectors. This allows a traditional file system that uses block sectors to be used on the flash drive.

===Banks===
The FTL organizes data using structures called banks. When the FTL gets a request to write something to memory, it uses a bank list to determine which area of the drive should be used. Essentially a bank is a group of sequential addresses, that keeps track of when it was last updated using timestamps. The FTL will only write to that bank, and once there is not enough space to write anymore, it switches out the bank for the bank with the oldest timestamp.

=Questions=

=References=

[1] Kim, Han-joon; Lee, Sang-goo. ''A New Flash Memory Management for Flash Storage System''. ''IEEExplore''. Dept. of Comput. Sci., Seoul Nat. Univ., 06 Aug 2002. <http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=812717&tag=1#>

[2] Smith, Lance. ''NAND Flash Solid State Storage Performance and Capability''. ''Flash Memory Summit''. SNIA Education Committee, 18 Aug 2009. <http://www.flashmemorysummit.com/English/Collaterals/Proceedings/2009/20090812_T1B_Smith.pdf>

[3] Chang, LiPin. ''On Efficient Wear Leveling for Large-Scale Flash-Memory Storage Systems''. ''Association for Computing Machinery (ACM)''. Dept. of Comput. Sci.,Nat. ChiaoTung Univ., 15 Mar 2007. <http://portal.acm.org/citation.cfm?id=1244248>

[4] Nath, Suman; Gibbons, Phillip. ''Online maintenance of very large random samples on flash storage''. ''Association for Computing Machinery (ACM)''. The VLDB Journal, 27 Jul 2007. <http://portal.acm.org/citation.cfm?id=1731355>

[5] Lim, Seung-Ho; Park; Kyu-Ho. ''An Efficient NAND Flash File System for Flash Memory Storage''. ''CORE Laboratory''. IEEE Transactions On Computers, Jul 2006. <http://vlsi.kaist.ac.kr/paper_list/2006_TC_CFFS.pdf>

[6] ''NAND vs. NOR Flash Memory Technology Overview''. ''RMG and Associates''. Toshiba America, accessed 14 Oct 2010. <http://maltiel-consulting.com/NAND_vs_NOR_Flash_Memory_Technology_Overview_Read_Write_Erase_speed_for_SLC_MLC_semiconductor_consulting_expert.pdf>

[7] Bez, Roberto; Camerlenghi, Emilio; Modelli, Alberto; Visconti, Angelo. ''Introduction to Flash Memory''. ''IEEExplore''. STMicroelectronics, 21 May 2003. <http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1199079&tag=1>

[8] Kawaguchi, Atsuo; Nishioka, Shingo; Motoda Hiroshi. ''A Flash-Memory Based File System''. ''CiteSeerX'' Advanced Research laboratory, Hitachi, Ltd., 1995. <http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.160.5142>

[9] Rosenblum, Mendel; Ouserhout, John. ''The Design and Implementation of a Log-structured File System''. ''Association for Computing Machinery (ACM)''. University of California at Berkeley, Feb 1992. <http://portal.acm.org/citation.cfm?id=146943&coll=GUIDE&dl=GUIDE&CFID=108397378&CFTOKEN=72657973&ret=1#Fulltext>

[10] Shimpi, Anand. ''Intel X25-M SSD: Intel Delivers One of the World's Fastest Drives''. ''AnAndTech''. AnAndTech, 8 Sep 2008. <http://www.anandtech.com/show/2614>

[11] Shimpi, Anand. ''The SSD Relapse: Understanding and Choosing the Best SSD''. ''AnAndTech''. AnAndTech, 30 Aug 2009. <http://www.anandtech.com/show/2829>

[12] Shimpi, Anand. ''The SSD Anthology: Understanding SSDs and New Drives from OCZ''. ''AnAndTech''. AnAndTech, 18 Mar 2009. <http://www.anandtech.com/show/2738>

[13] Corbet, Jonathan. ''Solid-State Storage Devices and the Block Layer''. ''Linux Weekly News''. Linux Weekly News, 4 Oct 2010. <http://lwn.net/Articles/408428/>

[14] Woodhouse, David. ''JFFS : The Journalling Flash File System''. ''CiteSeerX''. Red Hat, Inc, Accessed 14 Oct 2010. <http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.128.6156&rep=rep1&type=pdf>

[15] Agrawal, Nitin; Prabhakaran, Vijayan; Wobber, Ted; Davis, John; Manasse, Mark. Panigrahy, Rina. ''Design Tradeoffs for SSD Performance''. ''Association for Computing Machinery (ACM)'', USENIX 2008 Annual Technical Conference, 2008. <http://portal.acm.org/citation.cfm?id=1404014.1404019>

[16] Lee, Sang-Won, et al. ''A Log Buffer-Based Flash Translation Layer Using Fully-Associative Sector Translation''. ''Association for Computing Machinery (ACM)''. ACM Transactions on Embedded Computing Systems (TECS), Jul 2007. <http://portal.acm.org/citation.cfm?id=1275990>

[17] ''Reach New Heights in Computing Performance''. ''Micron Technology Inc''. Micro Technology Inc, Accessed 14 Oct 2010. <http://www.micron.com/products/solid_state_storage/client_ssd.html>

[18] ''Flash Memories.'' 1 ed. New York: Springer, 1999. Print.

[19] ''Nonvolatile Memory Technologies with Emphasis on Flash: A Comprehensive Guide to Understanding and Using Flash Memory Devices''. ''IEEE Press Series on Microelectronic Systems''. New York: Wiley-Ieee Press, 2008. Print.

[20] ''Nonvolatile Semiconductor Memory Technology: A Comprehensive Guide to Understanding and Using NVSM Devices''. ''IEEE Press Series on Microelectronic Systems''. New York: Wiley-Ieee Press, 1997. Print.

=External links=

Relevant Wikipedia articles: [http://en.wikipedia.org/wiki/Flash_Memory Flash Memory], [http://en.wikipedia.org/wiki/LogFS, LogFS], [http://en.wikipedia.org/wiki/Hard_disk Hard Disk Drives], [http://en.wikipedia.org/wiki/Wear_leveling Wear Leveling], [http://en.wikipedia.org/wiki/Hot_spot_%28computer_science%29 Hot Spots], [http://en.wikipedia.org/wiki/Solid-state_drive Sold State Drive].

=old=

Hey guys,

This is what I've got so far... mostly based on wikipedia:

Flash memory has two limitations: it can only be erased in blocks and and it wears out after a certain number of erase cycles. Furthermore, a particular kind of Flash memory (NAND) is not able to provide random access.
As a result of these Flash based file-systems cannot be handled in the same way as disk-based file systems. Here are a few of the key differences:

- Because memory must be erased in blocks, its erasure tends to take up time. Consequently, it is necessary to time the erasures in a way so as not to interfere with the efficiency of the system’s other operations. This is is not a real concern with disk-based file-systems.
- A disk file-system needs to minimize the seeking time, but Flash file-system does not concern itself with this as it doesn’t have a disk.
- A flash system tries to distribute memory in such a way so as not to make a particular block of memory subject to a disproportionally large number of erasures. The purpose of this is to keep the block from wearing out prematurely. The result of it is that memory needs to be distributed differently than in a disk based file-system.
Log-sturctured file systems are thus best suited to dealing with flash memory (they apparently do all of the above things).

For the essay form, I'm thinking of doing a section about traditional hard-disk systems, another about flash-memory and a third about flash systems. At this point, I am imagining the thesis as something like, "Flash systems require a fundamentally different system architecture than disk-based systems due to their need to adapt to the constraints inherent in flash memory: specifically, due to that memory's limited life-span and block-based erasures." The argument would then talk about how these two differences directly lead to a new FS approach.

That's how I see it at the moment. Honestly, I don't like doing research about this kind of stuff, so my data isn't very deep. That said, if you guys could find more info and summarize it, I'm pretty sure that I could synthesize it all into a coherent essay.

Fedor

COMP 3000 Essay 1 2010 Question 10

2010-10-15T01:55:31Z

Gsmith6: /* Flash Optimized File Systems */

COMP 3000 Essay 1 2010 Question 10

2010-10-15T00:19:53Z

Gsmith6:

=Question=

How do the constraints of flash storage affect the design of flash-optimized file systems? Explain by contrasting with hard disk-based file systems.

=Answer=
First introduced in the late 80s, Flash-memory is a light, energy-independent, compact, shock-resistent and efficiently readable type of storage. Because of the particular limitations of this kind of memory, flash file systems require a fundamentally different system architecture than disk-based file-systems: these systems need to be designed in light of flash-memory’s limited number of erase-cycles and its need to conduct erasures one entire block at a time. These constraints are a direct result of the same design that gives flash its advantages with regard to [ TO WHAT?] as both are due to [TO WHAT?] . Thus, a typical disk-based file-system is not suitable for working with flash memory as it erases far too frequently and indiscriminently while being simultaneously optimized for other constraints that do not affect flash memory. This means that a different solution is necessary and that solution is the log-based file-system which is far better suited to working with flash memory because it optimizes erasures by [WHAT?].

==Flash Memory==
Flash memory is non-volatile(meaning digital storage that does not require power to retain its memory) storage space that has become more popular recently due to its fast fetch times. There are two basic forms of the flash storage system, NOR and NAND. Each type has its advantages and disadvantages. NOR has the fastest read times, but is much slower at writing. NAND on the other hand has much more capacity, faster write times, is less expensive, and has a much longer life expectancy.[2]

More and more people use flash memory, with many sizes of drives, ranging from a few hundred megabyte USB key, to a few terrabyte internal solid-state drive(SSD). Two main reasons for this movement are because of flash's extremely fast read times, and its falling price. A typical flash drive has read speeds of up to 14 times faster than a hard disk drive (HDD).[17]

Although flash drives are exponentially faster than HDDs, they still have not become the main source of data management. The reason for this is because HDDs are simply much cheaper, and flash drives still have many faults. The most critical fault is that each block in flash memory can only be erased approximately 100,000 times.[14] This poses a problem because when modifying a file, even if its a single bit, the entire block must be erased, and rewritten. This erase/rewrite slows down the write operation considerably, making it actually slower to write a file to flash than an HDD.[8]

'''Possibly more on this later'''

HDDs use a block system, in which the kernel specifies which blocks to read and write. When using a flash drive, the blocks are emulated and mapped to a physical memory address. It does through what is called a "Transition Layer".

==Traditionally Optimized File Systems==

Since the kernel asks for a block number, a conventional hard disk drive (HDD) file-system is not optimized to work with flash memory. The reason for this is that conventional hard-disks have different constraints from those of flash memory - their primary problem is to reduce seeking time, while the primary problem when working with flash memory is to erase in a minimal and balanced way.

The most consuming process for an HDD is seeking data by relocating the read-head and spinning the magnetic disk. A traditional file system optimizes the way it stores data by placing related blocks close-by on the disk in order to minimize mechanical movement within the HDD. One of the great advantages of flash memory, which accounts for its fast read speed, is that there is no need to seek data physically. This is also why defragmentaion, a procedure used by HDDs to put files into more convenient configurations and thus minimize seeking times, loses its purpose in a flash memory context. Indeed, the unnecessary erasures that it entails are both inefficient and harmful for a flash memory unit.

This comes direclty out of flash memory's aforementioned constraints: the slow block-sized erasures and the limited number of erase-cycles. Because of these, a flash optimal file system needs to minimize its erase operations and also to spread out its erasures in such a way as to avoid the formation of hot-spots: sections of memory which have undergone a disproportionately high number of erasures and are thus in danger of burning out. This process of spreading out data is referred to as "wear leveling". To minimize hotspots, a system using flash memory would have to write new data to empty memory blocks. This method would also call for some sort of garbage collection to conduct necessary erasure operations while the system is idle. It makes better sense to do these at this time because of the slow nature of erasures in the flash memory context. Of course, there is no such feature in a traditional HDD file-system.

'''More on this later'''

==Flash Optimized File Systems==
''Flash Optimized files systems do to optimize HDD read/write/etc''

=Questions=

=References=

I'm going to start making up some proper citations for the websites we have. the order they're going in probably won't line up with the actual essay so I put the number for the link I'm actually citing next to each link, renumber as appropriate:

'''also, add in the right page number'''

[1] [http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=812717&tag=1#]

[2] [http://www.flashmemorysummit.com/English/Collaterals/Proceedings/2009/20090812_T1B_Smith.pdf]

[3][http://portal.acm.org/citation.cfm?id=1244248]

[4] [http://portal.acm.org/citation.cfm?id=1731355]

[5] [http://vlsi.kaist.ac.kr/paper_list/2006_TC_CFFS.pdf]

[6] [http://maltiel-consulting.com/NAND_vs_NOR_Flash_Memory_Technology_Overview_Read_Write_Erase_speed_for_SLC_MLC_semiconductor_consulting_expert.pdf]

[7] [http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1199079&tag=1]

[8] [http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.160.5142]

[9] [http://delivery.acm.org/10.1145/150000/146943/p26-rosenblum.pdf?key1=146943&key2=3656986821&coll=GUIDE&dl=GUIDE&CFID=108397378&CFTOKEN=72657973]

[10] [http://www.anandtech.com/show/2614]

[11] [http://www.anandtech.com/storage/showdoc.aspx?i=3531&p=1]

[12] [http://anandtech.com/storage/showdoc.aspx?i=3631]

[13] [http://lwn.net/SubscriberLink/408428/68fa8465da45967a/]

[14] [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.128.6156&rep=rep1&type=pdf]

[15] [http://portal.acm.org/citation.cfm?id=1404014.1404019]

[16] [http://portal.acm.org/citation.cfm?id=1275990]

[17] [http://www.micron.com/products/solid_state_storage/client_ssd.html]

==actual references start here==

[1] Kim, Han-joon; Lee, Sang-goo. ''A New Flash Memory Management for Flash Storage System''. ''IEEExplore''. Dept. of Comput. Sci., Seoul Nat. Univ., 06 Aug 2002. <http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=812717&tag=1#>

[2] Smith, Lance. ''NAND Flash Solid State Storage Performance and Capability''. ''Flash Memory Summit''. SNIA Education Committee, 18 Aug 2009. <http://www.flashmemorysummit.com/English/Collaterals/Proceedings/2009/20090812_T1B_Smith.pdf>

[3] Chang, LiPin. ''On Efficient Wear Leveling for Large-Scale Flash-Memory Storage Systems''. ''Association for Computing Machinery (ACM)''. Dept. of Comput. Sci.,Nat. ChiaoTung Univ., 15 Mar 2007. <http://portal.acm.org/citation.cfm?id=1244248>

[4] Nath, Suman; Gibbons, Phillip. ''Online maintenance of very large random samples on flash storage''. ''Association for Computing Machinery (ACM)''. The VLDB Journal, 27 Jul 2007. <http://portal.acm.org/citation.cfm?id=1731355>

[5] Lim, Seung-Ho; Park; Kyu-Ho. ''An Efficient NAND Flash File System for Flash Memory Storage''. ''CORE Laboratory''. IEEE Transactions On Computers, Jul 2006. <http://vlsi.kaist.ac.kr/paper_list/2006_TC_CFFS.pdf>

[6] ''NAND vs. NOR Flash Memory Technology Overview''. ''RMG and Associates''. Toshiba America, accessed 14 Oct 2010. <http://maltiel-consulting.com/NAND_vs_NOR_Flash_Memory_Technology_Overview_Read_Write_Erase_speed_for_SLC_MLC_semiconductor_consulting_expert.pdf>

[7] Bez, Roberto; Camerlenghi, Emilio; Modelli, Alberto; Visconti, Angelo. ''Introduction to Flash Memory''. ''IEEExplore''. STMicroelectronics, 21 May 2003. <http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1199079&tag=1>

[8] Kawaguchi, Atsuo; Nishioka, Shingo; Motoda Hiroshi. ''A Flash-Memory Based File System''. ''CiteSeerX'' Advanced Research laboratory, Hitachi, Ltd., 1995. <http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.160.5142>

[9] Rosenblum, Mendel; Ouserhout, John. ''The Design and Implementation of a Log-structured File System''. ''Association for Computing Machinery (ACM)''. University of California at Berkeley, Feb 1992. <http://portal.acm.org/citation.cfm?id=146943&coll=GUIDE&dl=GUIDE&CFID=108397378&CFTOKEN=72657973&ret=1#Fulltext>

[10] Shimpi, Anand. ''Intel X25-M SSD: Intel Delivers One of the World's Fastest Drives''. ''AnAndTech''. AnAndTech, 8 Sep 2008. <http://www.anandtech.com/show/2614>

[11] Shimpi, Anand. ''The SSD Relapse: Understanding and Choosing the Best SSD''. ''AnAndTech''. AnAndTech, 30 Aug 2009. <http://www.anandtech.com/show/2829>

[12] Shimpi, Anand. ''The SSD Anthology: Understanding SSDs and New Drives from OCZ''. ''AnAndTech''. AnAndTech, 18 Mar 2009. <http://www.anandtech.com/show/2738>

[13] Corbet, Jonathan. ''Solid-State Storage Devices and the Block Layer''. ''Linux Weekly News''. Linux Weekly News, 4 Oct 2010. <http://lwn.net/Articles/408428/>

[14] Woodhouse, David. ''JFFS : The Journalling Flash File System''. ''CiteSeerX''. Red Hat, Inc, accessed 14 Oct 2010. <http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.128.6156&rep=rep1&type=pdf>

[15] Agrawal, Nitin; Prabhakaran, Vijayan; Wobber, Ted; Davis, John; Manasse, Mark. Panigrahy, Rina. ''Design Tradeoffs for SSD Performance''. ''Association for Computing Machinery (ACM)'', USENIX 2008 Annual Technical Conference, 2008. <http://portal.acm.org/citation.cfm?id=1404014.1404019>

[16] Lee, Sang-Won, et al. ''A Log Buffer-Based Flash Translation Layer Using Fully-Associative Sector Translation''. ''Association for Computing Machinery (ACM)''. ACM Transactions on Embedded Computing Systems (TECS), Jul 2007. <http://portal.acm.org/citation.cfm?id=1275990>

[17] Micron Technology Inc, ''Reach New Heights in Computing Performance''. <http://www.micron.com/products/solid_state_storage/client_ssd.html> (may need reworking)

[18] Flash Memories. 1 ed. New York: Springer, 1999. Print.

[19] Nonvolatile Memory Technologies with Emphasis on Flash: A Comprehensive Guide to Understanding and Using Flash Memory Devices (IEEE Press Series on Microelectronic Systems). New York: Wiley-Ieee Press, 2008. Print.

[20] Nonvolatile Semiconductor Memory Technology: A Comprehensive Guide to Understanding and Using NVSM Devices (IEEE Press Series on Microelectronic Systems). New York: Wiley-Ieee Press, 1997. Print.

=External links=

Relivant Wikipedia articles: [http://en.wikipedia.org/wiki/Flash_Memory Flash Memory], [http://en.wikipedia.org/wiki/LogFS, LogFS], [http://en.wikipedia.org/wiki/Hard_disk Hard Disk Drives], [http://en.wikipedia.org/wiki/Wear_leveling Wear Leveling], [http://en.wikipedia.org/wiki/Hot_spot_%28computer_science%29 Hot Spots], [http://en.wikipedia.org/wiki/Solid-state_drive Sold State Drive].

=old=

Hey guys,

This is what I've got so far... mostly based on wikipedia:

Flash memory has two limitations: it can only be erased in blocks and and it wears out after a certain number of erase cycles. Furthermore, a particular kind of Flash memory (NAND) is not able to provide random access.
As a result of these Flash based file-systems cannot be handled in the same way as disk-based file systems. Here are a few of the key differences:

- Because memory must be erased in blocks, its erasure tends to take up time. Consequently, it is necessary to time the erasures in a way so as not to interfere with the efficiency of the system’s other operations. This is is not a real concern with disk-based file-systems.
- A disk file-system needs to minimize the seeking time, but Flash file-system does not concern itself with this as it doesn’t have a disk.
- A flash system tries to distribute memory in such a way so as not to make a particular block of memory subject to a disproportionally large number of erasures. The purpose of this is to keep the block from wearing out prematurely. The result of it is that memory needs to be distributed differently than in a disk based file-system.
Log-sturctured file systems are thus best suited to dealing with flash memory (they apparently do all of the above things).

For the essay form, I'm thinking of doing a section about traditional hard-disk systems, another about flash-memory and a third about flash systems. At this point, I am imagining the thesis as something like, "Flash systems require a fundamentally different system architecture than disk-based systems due to their need to adapt to the constraints inherent in flash memory: specifically, due to that memory's limited life-span and block-based erasures." The argument would then talk about how these two differences directly lead to a new FS approach.

That's how I see it at the moment. Honestly, I don't like doing research about this kind of stuff, so my data isn't very deep. That said, if you guys could find more info and summarize it, I'm pretty sure that I could synthesize it all into a coherent essay.

Fedor

Talk:COMP 3000 Essay 1 2010 Question 10

2010-10-14T23:51:45Z

Gsmith6:

Hey all,

I think we should write down our emails here so we can further discuss stuff without having to login here.
('''***Note that discussions over email can't be counted towards your participation grade!***'''--[[User:Soma|Anil]])

Geoff Smith (gsmith0413@gmail.com) - gsmith6

Andrew Bujáki (abujaki [at] Connect or Live.ca)
***I'm usually on MSN(Live) for collaboration at nights, Just make sure to put in a little message about who you are when you're adding me. :)

I used Google Scholar and came to this page http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=812717&tag=1#
Which briefly touches on the issues of Flash memory. Specifically, inability to update in place, and limited write/erase cycles.

Inability to update in place could refer to the way the flash disk is programmed, instead of bit-by-bit, it is programmed block-by-block. A block would have to be erased and completely reprogrammed in order to flip one bit after it's been set.
http://en.wikipedia.org/wiki/Flash_memory#Block_erasure

Limited write/erase: Flash memory typically has a short lifespan if it's being used a lot. Writing and erasing the memory (Changing, updating, etc) Will wear it out. Flash memory has a finite amount of writes, (varying on manufacturer, models, etc), and once they've been used up, you'll get bad sectors, corrupt data, and generally be SOL.
http://en.wikipedia.org/wiki/Flash_memory#Memory_wear

Filesystems would have to be changed to play nicely with these constraints, where it must use blocks efficiently and nicely, and minimize writing/erasing as much as possible.

I found a paper that talks about the performance, capabilities and limitations of NAND flash storage.

Abstract: "This presentation provides an in-depth examination of the
fundamental theoretical performance, capabilities, and
limitations of NAND Flash-based Solid State Storage (SSS). The
tutorial will explore the raw performance capabilities of NAND
Flash, and limitations to performance imposed by mitigation of
reliability issues, interfaces, protocols, and technology types.
Best practices for system integration of SSS will be discussed.
Performance achievements will be reviewed for various
products and applications. "

Link: http://www.flashmemorysummit.com/English/Collaterals/Proceedings/2009/20090812_T1B_Smith.pdf

There's no Starting place like Wikipedia, even if you shouldn't source it.

http://en.wikipedia.org/wiki/Flash_Memory

http://en.wikipedia.org/wiki/LogFS

http://en.wikipedia.org/wiki/Hard_disk

http://en.wikipedia.org/wiki/Wear_leveling

http://en.wikipedia.org/wiki/Hot_spot_%28computer_science%29

http://en.wikipedia.org/wiki/Solid-state_drive

Hey Guys,

We really don't have much time to get this done. Lets meet tomorrow after class and get our bearings to do this properly.

Fedor

A few of us have Networking immediately after class. I know personally I won't be able to make anything set on Tuesday.
Additionally, he spoke briefly about hotspots on the disk for our question last week, where places on the disk would be written to far more often than others.
As well, for bibliographical citing, http://bibme.org is a wonderful resource for the popular formats (I.e. MLA). If it should come down to that.
~Andrew

===links===

Start Posting some stuff to source from:

http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1199079&tag=1
--"Introduction to flash memory"

http://portal.acm.org/citation.cfm?id=1244248
--"Wear Leveling" (it's about a proposed way of doing it, but explains a whole bunch of other things to do that)

http://portal.acm.org/citation.cfm?id=1731355
--"Online maintenance of very large random samples on flash storage" (ie dealing with the constraints of Flash Storage in a system that might actually be written to 100000 times)

http://vlsi.kaist.ac.kr/paper_list/2006_TC_CFFS.pdf
--"An Efficient NAND Flash File System for Flash Memory Storage" discuses shortcomings of using hard disk based file systems and current flash based file systems

http://maltiel-consulting.com/NAND_vs_NOR_Flash_Memory_Technology_Overview_Read_Write_Erase_speed_for_SLC_MLC_semiconductor_consulting_expert.pdf
--"NAND vs NOR Flash Memory" (note: i didn't get this off of Google scholar but it seems to be written by someone from Toshiba. is that ok?)

Hi everybody,

So here are the latest news. Geoff, Andrew and myself had a meeting after class today and came up with a plan for writing this thing.

We decided to have 3 parts:

1. What flash storage is, why its good but also why it must have the problems that it does (the assumption is that it must have them, why would it otherwise?)
[don't know much about this just now... basics include that there is NOR (reads slightly faster)and NAND (holds more, writes faster, erases much faster, lasts about ten times longer) flash with NAND being especially popular for storage (what's NOR good for?). Here, we'd ideally want to talk about why flash was invented (supposed as an alternative to slow ROM), why it was suitable for that, and how it works on a technical level. Then, we'd want to mention why this technical functionality was innovative and useful but also why it came with two serious set-backs: having a limited-number of re-write cycles and needing to erase a block at a time.]

Either way, Flash storage affords far faster fetch times than the traditional platter-based HDD, and stability of information in a sense. Where the data is not actually stored, but reprogrammed, in a sense, the data is more secure and is less likely to be erased easily. On that note, in order to flip a single bit, that entire block will need to be erased, then reprogrammed. In an 'old' HDD, let's say, When the HDD fails at the end of its life cycle, your data is gone. (unless you're willing to shell out $200/hr to have it recovered, yes I've seen companies in Ottawa that do this.) In a flash HDD, when it reaches the end of its life, it merely becomes read-only. Bugger for Databases, but useful for technical notes and archives, let's say.
With today's modern gaming computers, Flash memory can be good on quick load times, however with limited read-writes, it could afford better use to things that are not updated as frequently. I.e... Well I don't have a better example than a webserver hosting a company's CSS and scripts.
~Source: Years in the 'biz

Flash memory started out as a replacement for EPROMs. At the time EPROMs needed a UV photoemission to be erased while flash memory could be erased electronically. The first flash memory product came out in 1988 but it did not take off until the late 1990’s because it could not be reliable produced. NOR and NAND memory is named after the arrangement of the cells in the memory array. NOR based flash memory benefits from having very fast burst read times but slower write times. Due to the structure of NOR memory programs stored in NOR based memory can be executed without being loaded into RAM first. NAND flash memory has a very large storage capacity and can read and write large files relatively fast. NAND is more suited for storage while NOR memory is better suited for direct program execution such as in CMOS chips.
source: http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1199079&tag=1 , http://maltiel-consulting.com/NAND_vs_NOR_Flash_Memory_Technology_Overview_Read_Write_Erase_speed_for_SLC_MLC_semiconductor_consulting_expert.pdf

2. How a traditional disk-based file-system works and why the limitations of flash storage make the two a poor match
[the obvious answer seems to be that traditional file-systems could just write to whatever memory was available but if they did this with a flash file-systems, certain chunks of memory would become unusable before others and the memory would be more difficult to work with. Also, disk-based file systems need to deal with seeking times which means that they want to organize their data in such a way as to reduce those (by putting related things together?) - with Flash, this isn't really a problem and thus one constraint the less to be concerned with.]

3. How a log based file-system works and why this method of operation is so well suited to working with flash memory especially in light of the latter's inherent limitations
[...]

At this time, the plan is that Geoff will work on #3 today, Andrew will work on #1 tomorrow and I will work on #2 tomorrow. The three of us will make an effort to consult some somewhat more painfully technical literature in order to gain insight into our respective queries. Whatever insight we find will be posted here.

Then, we will meet again on Thursday after class to decide how to actually write the essay.

PS, if there is anybody in the group besides the three of us - let us know so you can find a way to contribute to this... as at least two of us are competent essayists, painfully technical research would on one or more of the above topics would be a great way to contribute... especially if you could post it here prior to one of us going over the same thing.

Fedor

-- I'm not that great (but absolutely horrid) at essays and I'm alright at research, but if nothing else I have Thursday off and nothing (else) that needs doing by Friday so I can probably spend a bunch of time working on it just before it's due. -- ''Nick L''

-- Hay sorry I was unable to attend the meeting after class today. I am not too good at writing essays as well but I am pretty good at summarizing and researching. I am not too sure at what you would like me to do. Right now I'll assume you need me to research/summarizing articles for the 3 topics above. If you need me to do anything else post it here. I'll be checking the discussion regularly until this due. once again sorry for missing the meeting-- Paul Cox.

-- Hey i'm also supposed to be in on this. Sorry i couldn't contribute sooner because i was playing catchup in my other classes. Let me know what i can do and i'll be on it asap. - kirill (k.kashigin@gmail.com)
update: i'm gonna be helping Fedor with #2

PS, this article http://docs.google.com/viewer?a=v&q=cache:E7-H_pv_18wJ:citeseerx.ist.psu.edu/viewdoc/download%3Fdoi%3D10.1.1.92.2279%26rep%3Drep1%26type%3Dpdf+flash+memory+and+disk-based+file+systems&hl=en&gl=ca&pid=bl&srcid=ADGEESgspy-jqIdLOpaLYlPPoM56kjLPwXcL3_eMbTTBRkI7PG0jQKl9vIieTAYHubPu0EdQ0V4ccaf_p0S_SnqKMirSIM0Qoq5E0NpLd0M7LAGaE51wkD0F55cRSkX8dnTqx_9Yx2E7&sig=AHIEtbS-yfGI9Y48DJ0WyEEhmsXInelRGw looks really useful for part 3.

---same article as above but shorter link: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.160.5142

PPS, and this article looks really great for understanding how log based file systems work: http://delivery.acm.org/10.1145/150000/146943/p26-rosenblum.pdf?key1=146943&key2=3656986821&coll=GUIDE&dl=GUIDE&CFID=108397378&CFTOKEN=72657973

Hey Luc (TA) here, Anandtech ran a series of articles on solid state drives that you guys might find useful. It mostly looked at hardware aspects but it gives some interesting insights on how to modify file systems to better support flash memory.

http://www.anandtech.com/cpuchipsets/intel/showdoc.aspx?i=3403

http://www.anandtech.com/storage/showdoc.aspx?i=3531&p=1

http://anandtech.com/storage/showdoc.aspx?i=3631

--[[User:3maisons|3maisons]] 19:44, 12 October 2010 (UTC)

Hey Paul&Kirill,

If one of you guys could help me out with #2, that would be really great. I was going to work on that tomorrow, but I also have another large assignment to deal with and not having to do this research would greatly ease my life. Moreover, I do intend to work on writing&polishing the essay on Thursday as I have a lot of experience with that and it far more than research. Let me know if either one of you can help me with this.

The other person could probably read over what Luc posted for us and see if it fits into our framework. Just be sure to state who is going to do what.

Nick,

Honestly, we really hope to have the research done by Thursday. If that is the only day that you are free and you're not a writer, I'm honestly not sure what you could do. Perhaps someone else can think of something.

- Fedor

I'm gonna have something for #2 up tonight. -kirill

So I found this article on Reddit, posted from Linux Weekly News on pretty much exactly what we are looking at. It's entitled "Solid-state storage devices and the block layer"

http://lwn.net/SubscriberLink/408428/68fa8465da45967a/ --[[User:Gsmith6|Gsmith6]] 20:36, 13 October 2010 (UTC)

I wasn't exactly sure how much information i was supposed to present but here's what i got for #2:

Most conventional file systems are designed to me implemented on hard disk drives. This fact does not mean they cannot be implemented on a solid state drive (file storage that uses flash memory instead of magnetic discs). It would however, in many ways, defeat the purpose of using flash memory. The most consuming process for an HDD is seeking data by relocating the read-head and spinning the magnetic disk. A traditional file system optimizes the way it stores data by placing related blocks close-by on the disk to minimize mechanical movement within the HDD. One of the great advantages of flash memory, which accounts for its fast read speed, is that there is no need to seek data physically so there is no need to waste resources laying out the data in close proximity.
A traditional HDD file system will also attempt to defragment itself, moving blocks of data around for closer proximity on the magnetic disk. This process, although beneficial for HDD's, is harmful and inefficient for flash based storage. A flash optimal file system needs to reduce the amount of erase operations, since flash memory only has a limited amount of erase cycles as well as having very slow erase speeds.
When an HDD rewrites data to a physical location there is no need for it to erase the previously occupying data first, so a traditional disk based file system doesn't worry about erasing data from unused memory blocks. In contrast flash memory needs to first erase the data block before it can modify any of it contents. Since the erase procedure is extremely slow, its not practical to overwrite old data every time. It is also decremental to the life span of flash memory.
To maximize the potential of flash based memory the file system would have to write new data to empty memory blocks. This method would also call for some sort of garbage collection to erase unused blocks when the system is idle, which does not get implemented in conventional file systems since it is not needed.

--kirill

So Fedor and I were talking in the labs, and we came to the conclusion that we have been focusing on just the translation from a regular file system to a flash drive. We were under the impression that this was in fact the "Flash Optimized System", but pulling up some more articles, I'm finding that this is not necessarily the case.

This paper here http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.128.6156&rep=rep1&type=pdf.
shows an example from Axis Communications where they developed a file system specifically designed to be used on flash drives.

Now I haven't completely read it, so it might just be an optimized translational system, but its at least a start.

At our meeting today, we've decided that it would be best if people could post a rough summary of their notes in the appropriate sections, and I will rewrite them into an essay, which Fedor will go through later tonight to edit and add some more information.

Paul: I missed your comment that you weren't that great at writing, and good at research. If you want some articles behind the pay-walls, I've saved a bunch of them and emailed them to myself. Just email me (at the address at the top of the page) and I'll be more than happy to send some your way.

PS. some more references

Design tradeoffs for SSD performance http://portal.acm.org/citation.cfm?id=1404014.1404019
A log buffer-based flash translation layer using fully-associative sector translation http://delivery.acm.org/10.1145/1280000/1275990/a18-lee.pdf?key1=1275990&key2=0709607821&coll=GUIDE&dl=GUIDE&CFID=105787273&CFTOKEN=74601780

--[[User:Gsmith6|Gsmith6]] 15:03, 14 October 2010 (UTC)

I don't have any notes on this computer. >: I will be adding more to my section later on tonight. Sorry. ~Andrew

Hello dudes,

Just a quick note, try to include citations in your paragraphs - each time that you make a claim which came from evidence, put a little number [X, pp. page-number (if applicable)] into your text. Then, put the same [X] at the bottom of the page with the bibliographical information about the source. The prof hasn't yet gotten back to me about his preferred citation format, so just stick with this one for now:

Authors. ''Title''. Web-page. Date of article. Web (the word). Date you accessed it.

Here's an example:

[1] Kawaguchi, Nishioka, Tamoda. ''A Flash Memory Based File System''.
http://docs.google.com/viewer?a=v&q=cache:E7-H_pv_18wJ:citeseerx.ist.psu.edu/viewdoc/download%3Fdoi%3D10.1.1.92.2279%26rep%3Drep1%26type%3Dpdf+flash+memory+and+disk-based+file+systems&hl=en&gl=ca&pid=bl&srcid=ADGEESgspy-jqIdLOpaLYlPPoM56kjLPwXcL3_eMbTTBRkI7PG0jQKl9vIieTAYHubPu0EdQ0V4ccaf_p0S_SnqKMirSIM0Qoq5E0NpLd0M7LAGaE51wkD0F55cRSkX8dnTqx_9Yx2E7&sig=AHIEtbS-yfGI9Y48DJ0WyEEhmsXInelRGw . 1995. Web. Oct. 14, 2010.

Fedor

PS, its a good idea to check this fairly frequently between now and tomorrow morning - you never know when something will come up.

Phew... for a while there I was starting to think that I had nothing about the actual "Log-Based System", but it turns out that the "Transitional Layer" is the same thing. It looks like some articles are calling it the Log system, while others are calling it the transitional layer. Pretty sure I'm going to have an experts knowledge about flash drives after reading all these articles :P
--[[User:Gsmith6|Gsmith6]] 18:13, 14 October 2010 (UTC)

Hay Geoff this is what i got so far after reading a couple of the pdfs. the double tabed points are just my annotation on how they relate to the question.

Ware leveling: p1126-chang.pdf

* Uneven wearing of flash memory due to storing data close together
* Garbage collection prefers that no blocks have pages that have data that is constantly becoming invalid
* data that remains the same for longs periods of time should be moved from block that have not be written to much and moved to blocks that haven been erased frequently.

Log file structure: 926-rosenblum.pdf

* LFS based on assumption that frequently read files will be stored in cash and that the hard disk traffic will be dominated by writes
* Writes all new info to disk in a sequential structure called a log
* Data is stored permanently in these logs no other data is stored on the hard drive
* Converts many small random synchronous writes to a large asynchronous sequential write
** Good for flash because it cuts down on writing (prolongs drive life)
** It also good because it writes to a bigger section then a page. This means it can fill a block at a time so it doesn’t fill up other blocks with random writes that would later need to be cleaned. Cuts down on cleaning.
* Inode is stored in the log on the disk while an inode map is maintained in memory which points to the inode in the hard disk. as
** This is good for flash drives because reading does not hurt the drives life and it is fast.
** This means the map will not have to be updated on the disk as frequently cutting down on the writes.
* Log systems weakness is that it is susceptible to becoming fragmented due to the larger writes.
** Since flash drives do not require fragmentation this is fine. Also since flash drives have very fast random access the system does not become bogged down when the logs are fragmented
* Log system implements a cleaning system that scans a segment in and sees if there is live data in it. If there is a certain percentage of invalid data which goes according to the cleaning policy it will be cleaned. All the live data will be copied out and the segment will be erased.
** This is a garbage collector but its built into the file system.
* Segments contain a number of blocks. Segments can contain logs or parts of logs.

I'm still sifting though the other pdfs you sent me. Would you like me to post more info or should i format this into a paragraph or 2?

--Paul

I'm sorting through my notes and putting them into a digital format. Difficult because It's hard to remember what I copied directly out of the notes for my own reference, so I'm trying to de-plagarize. I'll upload what I have to the section by 20:00, but expect more updates. ~Andrew

Hey Paul,

Thanks for helping out with the research
I'm thinking that we probably have enough information (at least for the log-based system), so maybe if you could "try" and putting them into paragraphs. I'm kind of in the middle of typing up the essay, and I'm uploading new information every 10-20 minutes or so. Even if you could get something into paragraph form, I can go through and edit, rearrange, and add stuff as I get there. --[[User:Gsmith6|Gsmith6]] 23:51, 14 October 2010 (UTC)

Talk:COMP 3000 Essay 1 2010 Question 10

2010-10-14T23:51:31Z

Gsmith6:

Hey all,

I think we should write down our emails here so we can further discuss stuff without having to login here.
('''***Note that discussions over email can't be counted towards your participation grade!***'''--[[User:Soma|Anil]])

Geoff Smith (gsmith0413@gmail.com) - gsmith6

Andrew Bujáki (abujaki [at] Connect or Live.ca)
***I'm usually on MSN(Live) for collaboration at nights, Just make sure to put in a little message about who you are when you're adding me. :)

I used Google Scholar and came to this page http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=812717&tag=1#
Which briefly touches on the issues of Flash memory. Specifically, inability to update in place, and limited write/erase cycles.

Inability to update in place could refer to the way the flash disk is programmed, instead of bit-by-bit, it is programmed block-by-block. A block would have to be erased and completely reprogrammed in order to flip one bit after it's been set.
http://en.wikipedia.org/wiki/Flash_memory#Block_erasure

Limited write/erase: Flash memory typically has a short lifespan if it's being used a lot. Writing and erasing the memory (Changing, updating, etc) Will wear it out. Flash memory has a finite amount of writes, (varying on manufacturer, models, etc), and once they've been used up, you'll get bad sectors, corrupt data, and generally be SOL.
http://en.wikipedia.org/wiki/Flash_memory#Memory_wear

Filesystems would have to be changed to play nicely with these constraints, where it must use blocks efficiently and nicely, and minimize writing/erasing as much as possible.

I found a paper that talks about the performance, capabilities and limitations of NAND flash storage.

Abstract: "This presentation provides an in-depth examination of the
fundamental theoretical performance, capabilities, and
limitations of NAND Flash-based Solid State Storage (SSS). The
tutorial will explore the raw performance capabilities of NAND
Flash, and limitations to performance imposed by mitigation of
reliability issues, interfaces, protocols, and technology types.
Best practices for system integration of SSS will be discussed.
Performance achievements will be reviewed for various
products and applications. "

Link: http://www.flashmemorysummit.com/English/Collaterals/Proceedings/2009/20090812_T1B_Smith.pdf

There's no Starting place like Wikipedia, even if you shouldn't source it.

http://en.wikipedia.org/wiki/Flash_Memory

http://en.wikipedia.org/wiki/LogFS

http://en.wikipedia.org/wiki/Hard_disk

http://en.wikipedia.org/wiki/Wear_leveling

http://en.wikipedia.org/wiki/Hot_spot_%28computer_science%29

http://en.wikipedia.org/wiki/Solid-state_drive

Hey Guys,

We really don't have much time to get this done. Lets meet tomorrow after class and get our bearings to do this properly.

Fedor

A few of us have Networking immediately after class. I know personally I won't be able to make anything set on Tuesday.
Additionally, he spoke briefly about hotspots on the disk for our question last week, where places on the disk would be written to far more often than others.
As well, for bibliographical citing, http://bibme.org is a wonderful resource for the popular formats (I.e. MLA). If it should come down to that.
~Andrew

===links===

Start Posting some stuff to source from:

http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1199079&tag=1
--"Introduction to flash memory"

http://portal.acm.org/citation.cfm?id=1244248
--"Wear Leveling" (it's about a proposed way of doing it, but explains a whole bunch of other things to do that)

http://portal.acm.org/citation.cfm?id=1731355
--"Online maintenance of very large random samples on flash storage" (ie dealing with the constraints of Flash Storage in a system that might actually be written to 100000 times)

http://vlsi.kaist.ac.kr/paper_list/2006_TC_CFFS.pdf
--"An Efficient NAND Flash File System for Flash Memory Storage" discuses shortcomings of using hard disk based file systems and current flash based file systems

http://maltiel-consulting.com/NAND_vs_NOR_Flash_Memory_Technology_Overview_Read_Write_Erase_speed_for_SLC_MLC_semiconductor_consulting_expert.pdf
--"NAND vs NOR Flash Memory" (note: i didn't get this off of Google scholar but it seems to be written by someone from Toshiba. is that ok?)

Hi everybody,

So here are the latest news. Geoff, Andrew and myself had a meeting after class today and came up with a plan for writing this thing.

We decided to have 3 parts:

1. What flash storage is, why its good but also why it must have the problems that it does (the assumption is that it must have them, why would it otherwise?)
[don't know much about this just now... basics include that there is NOR (reads slightly faster)and NAND (holds more, writes faster, erases much faster, lasts about ten times longer) flash with NAND being especially popular for storage (what's NOR good for?). Here, we'd ideally want to talk about why flash was invented (supposed as an alternative to slow ROM), why it was suitable for that, and how it works on a technical level. Then, we'd want to mention why this technical functionality was innovative and useful but also why it came with two serious set-backs: having a limited-number of re-write cycles and needing to erase a block at a time.]

Either way, Flash storage affords far faster fetch times than the traditional platter-based HDD, and stability of information in a sense. Where the data is not actually stored, but reprogrammed, in a sense, the data is more secure and is less likely to be erased easily. On that note, in order to flip a single bit, that entire block will need to be erased, then reprogrammed. In an 'old' HDD, let's say, When the HDD fails at the end of its life cycle, your data is gone. (unless you're willing to shell out $200/hr to have it recovered, yes I've seen companies in Ottawa that do this.) In a flash HDD, when it reaches the end of its life, it merely becomes read-only. Bugger for Databases, but useful for technical notes and archives, let's say.
With today's modern gaming computers, Flash memory can be good on quick load times, however with limited read-writes, it could afford better use to things that are not updated as frequently. I.e... Well I don't have a better example than a webserver hosting a company's CSS and scripts.
~Source: Years in the 'biz

Flash memory started out as a replacement for EPROMs. At the time EPROMs needed a UV photoemission to be erased while flash memory could be erased electronically. The first flash memory product came out in 1988 but it did not take off until the late 1990’s because it could not be reliable produced. NOR and NAND memory is named after the arrangement of the cells in the memory array. NOR based flash memory benefits from having very fast burst read times but slower write times. Due to the structure of NOR memory programs stored in NOR based memory can be executed without being loaded into RAM first. NAND flash memory has a very large storage capacity and can read and write large files relatively fast. NAND is more suited for storage while NOR memory is better suited for direct program execution such as in CMOS chips.
source: http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1199079&tag=1 , http://maltiel-consulting.com/NAND_vs_NOR_Flash_Memory_Technology_Overview_Read_Write_Erase_speed_for_SLC_MLC_semiconductor_consulting_expert.pdf

2. How a traditional disk-based file-system works and why the limitations of flash storage make the two a poor match
[the obvious answer seems to be that traditional file-systems could just write to whatever memory was available but if they did this with a flash file-systems, certain chunks of memory would become unusable before others and the memory would be more difficult to work with. Also, disk-based file systems need to deal with seeking times which means that they want to organize their data in such a way as to reduce those (by putting related things together?) - with Flash, this isn't really a problem and thus one constraint the less to be concerned with.]

3. How a log based file-system works and why this method of operation is so well suited to working with flash memory especially in light of the latter's inherent limitations
[...]

At this time, the plan is that Geoff will work on #3 today, Andrew will work on #1 tomorrow and I will work on #2 tomorrow. The three of us will make an effort to consult some somewhat more painfully technical literature in order to gain insight into our respective queries. Whatever insight we find will be posted here.

Then, we will meet again on Thursday after class to decide how to actually write the essay.

PS, if there is anybody in the group besides the three of us - let us know so you can find a way to contribute to this... as at least two of us are competent essayists, painfully technical research would on one or more of the above topics would be a great way to contribute... especially if you could post it here prior to one of us going over the same thing.

Fedor

-- I'm not that great (but absolutely horrid) at essays and I'm alright at research, but if nothing else I have Thursday off and nothing (else) that needs doing by Friday so I can probably spend a bunch of time working on it just before it's due. -- ''Nick L''

-- Hay sorry I was unable to attend the meeting after class today. I am not too good at writing essays as well but I am pretty good at summarizing and researching. I am not too sure at what you would like me to do. Right now I'll assume you need me to research/summarizing articles for the 3 topics above. If you need me to do anything else post it here. I'll be checking the discussion regularly until this due. once again sorry for missing the meeting-- Paul Cox.

-- Hey i'm also supposed to be in on this. Sorry i couldn't contribute sooner because i was playing catchup in my other classes. Let me know what i can do and i'll be on it asap. - kirill (k.kashigin@gmail.com)
update: i'm gonna be helping Fedor with #2

PS, this article http://docs.google.com/viewer?a=v&q=cache:E7-H_pv_18wJ:citeseerx.ist.psu.edu/viewdoc/download%3Fdoi%3D10.1.1.92.2279%26rep%3Drep1%26type%3Dpdf+flash+memory+and+disk-based+file+systems&hl=en&gl=ca&pid=bl&srcid=ADGEESgspy-jqIdLOpaLYlPPoM56kjLPwXcL3_eMbTTBRkI7PG0jQKl9vIieTAYHubPu0EdQ0V4ccaf_p0S_SnqKMirSIM0Qoq5E0NpLd0M7LAGaE51wkD0F55cRSkX8dnTqx_9Yx2E7&sig=AHIEtbS-yfGI9Y48DJ0WyEEhmsXInelRGw looks really useful for part 3.

---same article as above but shorter link: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.160.5142

PPS, and this article looks really great for understanding how log based file systems work: http://delivery.acm.org/10.1145/150000/146943/p26-rosenblum.pdf?key1=146943&key2=3656986821&coll=GUIDE&dl=GUIDE&CFID=108397378&CFTOKEN=72657973

Hey Luc (TA) here, Anandtech ran a series of articles on solid state drives that you guys might find useful. It mostly looked at hardware aspects but it gives some interesting insights on how to modify file systems to better support flash memory.

http://www.anandtech.com/cpuchipsets/intel/showdoc.aspx?i=3403

http://www.anandtech.com/storage/showdoc.aspx?i=3531&p=1

http://anandtech.com/storage/showdoc.aspx?i=3631

--[[User:3maisons|3maisons]] 19:44, 12 October 2010 (UTC)

Hey Paul&Kirill,

If one of you guys could help me out with #2, that would be really great. I was going to work on that tomorrow, but I also have another large assignment to deal with and not having to do this research would greatly ease my life. Moreover, I do intend to work on writing&polishing the essay on Thursday as I have a lot of experience with that and it far more than research. Let me know if either one of you can help me with this.

The other person could probably read over what Luc posted for us and see if it fits into our framework. Just be sure to state who is going to do what.

Nick,

Honestly, we really hope to have the research done by Thursday. If that is the only day that you are free and you're not a writer, I'm honestly not sure what you could do. Perhaps someone else can think of something.

- Fedor

I'm gonna have something for #2 up tonight. -kirill

So I found this article on Reddit, posted from Linux Weekly News on pretty much exactly what we are looking at. It's entitled "Solid-state storage devices and the block layer"

http://lwn.net/SubscriberLink/408428/68fa8465da45967a/ --[[User:Gsmith6|Gsmith6]] 20:36, 13 October 2010 (UTC)

I wasn't exactly sure how much information i was supposed to present but here's what i got for #2:

Most conventional file systems are designed to me implemented on hard disk drives. This fact does not mean they cannot be implemented on a solid state drive (file storage that uses flash memory instead of magnetic discs). It would however, in many ways, defeat the purpose of using flash memory. The most consuming process for an HDD is seeking data by relocating the read-head and spinning the magnetic disk. A traditional file system optimizes the way it stores data by placing related blocks close-by on the disk to minimize mechanical movement within the HDD. One of the great advantages of flash memory, which accounts for its fast read speed, is that there is no need to seek data physically so there is no need to waste resources laying out the data in close proximity.
A traditional HDD file system will also attempt to defragment itself, moving blocks of data around for closer proximity on the magnetic disk. This process, although beneficial for HDD's, is harmful and inefficient for flash based storage. A flash optimal file system needs to reduce the amount of erase operations, since flash memory only has a limited amount of erase cycles as well as having very slow erase speeds.
When an HDD rewrites data to a physical location there is no need for it to erase the previously occupying data first, so a traditional disk based file system doesn't worry about erasing data from unused memory blocks. In contrast flash memory needs to first erase the data block before it can modify any of it contents. Since the erase procedure is extremely slow, its not practical to overwrite old data every time. It is also decremental to the life span of flash memory.
To maximize the potential of flash based memory the file system would have to write new data to empty memory blocks. This method would also call for some sort of garbage collection to erase unused blocks when the system is idle, which does not get implemented in conventional file systems since it is not needed.

--kirill

So Fedor and I were talking in the labs, and we came to the conclusion that we have been focusing on just the translation from a regular file system to a flash drive. We were under the impression that this was in fact the "Flash Optimized System", but pulling up some more articles, I'm finding that this is not necessarily the case.

This paper here http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.128.6156&rep=rep1&type=pdf.
shows an example from Axis Communications where they developed a file system specifically designed to be used on flash drives.

Now I haven't completely read it, so it might just be an optimized translational system, but its at least a start.

At our meeting today, we've decided that it would be best if people could post a rough summary of their notes in the appropriate sections, and I will rewrite them into an essay, which Fedor will go through later tonight to edit and add some more information.

Paul: I missed your comment that you weren't that great at writing, and good at research. If you want some articles behind the pay-walls, I've saved a bunch of them and emailed them to myself. Just email me (at the address at the top of the page) and I'll be more than happy to send some your way.

PS. some more references

Design tradeoffs for SSD performance http://portal.acm.org/citation.cfm?id=1404014.1404019
A log buffer-based flash translation layer using fully-associative sector translation http://delivery.acm.org/10.1145/1280000/1275990/a18-lee.pdf?key1=1275990&key2=0709607821&coll=GUIDE&dl=GUIDE&CFID=105787273&CFTOKEN=74601780

--[[User:Gsmith6|Gsmith6]] 15:03, 14 October 2010 (UTC)

I don't have any notes on this computer. >: I will be adding more to my section later on tonight. Sorry. ~Andrew

Hello dudes,

Just a quick note, try to include citations in your paragraphs - each time that you make a claim which came from evidence, put a little number [X, pp. page-number (if applicable)] into your text. Then, put the same [X] at the bottom of the page with the bibliographical information about the source. The prof hasn't yet gotten back to me about his preferred citation format, so just stick with this one for now:

Authors. ''Title''. Web-page. Date of article. Web (the word). Date you accessed it.

Here's an example:

[1] Kawaguchi, Nishioka, Tamoda. ''A Flash Memory Based File System''.
http://docs.google.com/viewer?a=v&q=cache:E7-H_pv_18wJ:citeseerx.ist.psu.edu/viewdoc/download%3Fdoi%3D10.1.1.92.2279%26rep%3Drep1%26type%3Dpdf+flash+memory+and+disk-based+file+systems&hl=en&gl=ca&pid=bl&srcid=ADGEESgspy-jqIdLOpaLYlPPoM56kjLPwXcL3_eMbTTBRkI7PG0jQKl9vIieTAYHubPu0EdQ0V4ccaf_p0S_SnqKMirSIM0Qoq5E0NpLd0M7LAGaE51wkD0F55cRSkX8dnTqx_9Yx2E7&sig=AHIEtbS-yfGI9Y48DJ0WyEEhmsXInelRGw . 1995. Web. Oct. 14, 2010.

Fedor

PS, its a good idea to check this fairly frequently between now and tomorrow morning - you never know when something will come up.

Phew... for a while there I was starting to think that I had nothing about the actual "Log-Based System", but it turns out that the "Transitional Layer" is the same thing. It looks like some articles are calling it the Log system, while others are calling it the transitional layer. Pretty sure I'm going to have an experts knowledge about flash drives after reading all these articles :P
--[[User:Gsmith6|Gsmith6]] 18:13, 14 October 2010 (UTC)

Hay Geoff this is what i got so far after reading a couple of the pdfs. the double tabed points are just my annotation on how they relate to the question.

Ware leveling: p1126-chang.pdf

* Uneven wearing of flash memory due to storing data close together
* Garbage collection prefers that no blocks have pages that have data that is constantly becoming invalid
* data that remains the same for longs periods of time should be moved from block that have not be written to much and moved to blocks that haven been erased frequently.

Log file structure: 926-rosenblum.pdf

* LFS based on assumption that frequently read files will be stored in cash and that the hard disk traffic will be dominated by writes
* Writes all new info to disk in a sequential structure called a log
* Data is stored permanently in these logs no other data is stored on the hard drive
* Converts many small random synchronous writes to a large asynchronous sequential write
** Good for flash because it cuts down on writing (prolongs drive life)
** It also good because it writes to a bigger section then a page. This means it can fill a block at a time so it doesn’t fill up other blocks with random writes that would later need to be cleaned. Cuts down on cleaning.
* Inode is stored in the log on the disk while an inode map is maintained in memory which points to the inode in the hard disk. as
** This is good for flash drives because reading does not hurt the drives life and it is fast.
** This means the map will not have to be updated on the disk as frequently cutting down on the writes.
* Log systems weakness is that it is susceptible to becoming fragmented due to the larger writes.
** Since flash drives do not require fragmentation this is fine. Also since flash drives have very fast random access the system does not become bogged down when the logs are fragmented
* Log system implements a cleaning system that scans a segment in and sees if there is live data in it. If there is a certain percentage of invalid data which goes according to the cleaning policy it will be cleaned. All the live data will be copied out and the segment will be erased.
** This is a garbage collector but its built into the file system.
* Segments contain a number of blocks. Segments can contain logs or parts of logs.

I'm still sifting though the other pdfs you sent me. Would you like me to post more info or should i format this into a paragraph or 2?

--Paul

I'm sorting through my notes and putting them into a digital format. Difficult because It's hard to remember what I copied directly out of the notes for my own reference, so I'm trying to de-plagarize. I'll upload what I have to the section by 20:00, but expect more updates. ~Andrew

Hey Paul,

Thanks for helping out with the research
I'm thinking that we probably have enough information (at least for the log-based system), so maybe if you could "try" and putting them into paragraphs. I'm kind of in the middle of typing up the essay, and I'm uploading new information every 10-20 minutes or so. Even if you could get something into paragraph form, I can go through and edit, rearrange, and add stuff as I get there.