Soma-notes - User contributions [en]

Talk:COMP 3000 Essay 2 2010 Question 1

2010-12-03T03:42:12Z

Kkashigi: /* Conclusion: Sections 6 & 7 */

=Class and Notices=
(Nov. 30, 2010) Prof. Anil stated that we should focus on the 3 easiest to understand parts in section 5 and elaborate on them.

- Also, I, Daniel B., work Thursday night, so I will be finishing up as much of my part as I can for the essay before Thursday morning's class, then maybe we can all meet up in a lab in HP and put the finishing touches on the essay. I will be available online Wednesday night from about 6:30pm onwards and will be in the game dev lab or CCSS lounge Wednesday morning from about 11am to 2pm if anyone would like to meet up with me at those times.

- [[I suggest we meet up Thursday morning after Operating Systems in order to discuss and finalize the essay. Maybe we can even designate a lab for the group to meet up in. Any suggestions?]] - Daniel B.

- HP 3115 since there wont be a class in there (as its our tutorial and we know there won't be anyone there)
-- Go to Wireless Lab next to CCSS Lounge. Andrew and Dan B. will be there.

- If its all the same to you guys mind if I just join you via msn or iirc? Or phone if you really want. -Rannath

- I'm working today, but I'll be at a computer reading this page/contributing to my section. Depending on how busy I am, I should be able to get some significant writing in before 4pm today on my section and any additional sections required. RP

- I wont be there either. that does not mean i wont/cant contribute. I'll be on msn or you can just email me. -kirill

=Group members=
Patrick Young [mailto:Rannath@gmail.com Rannath@gmail.com]

Daniel Beimers [mailto:demongyro@gmail.com demongyro@gmail.com]

Andrew Bown [mailto:abown2@connect.carleton.ca abown2@connect.carleton.ca]

Kirill Kashigin [mailto:k.kashigin@gmail.com k.kashigin@gmail.com]

Rovic Perdon [mailto:rperdon@gmail.com rperdon@gmail.com]

Daniel Sont [mailto:dan.sont@gmail.com dan.sont@gmail.com]

=Methodology=
We should probably have our work verified by at least one group member before posting to the actual page

=To Do=
*The only thing we need to do is determine how fair the conclusion is.

*Thank God [[SIR DANIEL SONT OF OTTAWA, pay attention here]] I need you to take a look at the conclusion and help determine the "fairness" of it.

*We also need to make sure everything is moved over to the actual essay page, and once that is done, that all the references are done correctly.

=Essay=
==Paper - DONE!!!==
This paper was authored by - Silas Boyd-Wickizer, Austin T. Clements, Yandong Mao, Aleksey Pesterev, M. Frans Kaashoek, Robert Morris, and Nickolai Zeldovich.

They all work at MIT CSAIL.

The paper: [http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf An Analysis of Linux Scalability to Many Cores]

==Background Concepts - DONE!!!==
===memcached: ''Section 3.2''===
memcached is an in-memory hash table server. One instance of memcached running on many different cores is bottlenecked by an internal lock, which is avoided by the MIT team by running one instance per core. Clients each connect to a single instance of memcached, allowing the server to simulate parallelism without needing to make major changes to the application or kernel. With few requests, memcached spends 80% of its time in the kernel on one core, mostly processing packets.[1]

===Apache: ''Section 3.3''===
Apache is a web server that has been used in previous Linux scalability studies. In the case of this study, Apache has been configured to run a separate process on each core. Each process, in turn, has multiple threads (making it a perfect example of parallel programming). Each process uses one of their threads to accepting incoming connections and others are used to process these connections. On a single core processor, Apache spends 60% of its execution time in the kernel.[1]

===gmake: ''Section 3.5''===
gmake is an unofficial default benchmark in the Linux community which is used in this paper to build the Linux kernel. gmake takes a file called a makefile and processes its recipes for the requisite files to determine how and when to remake or recompile code. With a simple command -j or --jobs, gmake can process many of these recipes in parallel. Since gmake creates more processes than cores, it can make proper use of multiple cores to process the recipes.[2] Since gmake involves much reading and writing, in order to prevent bottlenecks due to the filesystem or hardware, the test cases use an in-memory filesystem tmpfs, which gives them a backdoor around the bottlenecks for testing purposes. In addition to this, gmake is limited in scalability due to the serial processes that run at the beginning and end of its execution, which limits its scalability to a small degree. gmake spends much of its execution time with its compiler, processing the recipes and recompiling code, but still spend 7.6% of its time in system time.[1]

[2] http://www.gnu.org/software/make/manual/make.html

==Research problem - DONE!!!==
As technology progresses, the number of core a main processor can have is increasing at an impressive rate. Soon personal computers will have so many cores that scalability will be an issue. There has to be a way that standard user level Linux kernel will scale with a 48-core system[[#Foot1|1]]. The problem with a standard Linux OS is that they are not designed for massive scalability, which will soon prove to be a problem. The issue with scalability is that a solo core will perform much more work compared to a single core working with 47 other cores. Although traditional logic states that the situation makes sense because there are 48 cores dividing the work, the information should be processed as fast as possible with each core doing as much work as possible.

To fix those scalability issues, it is necessary to focus on three major areas: the Linux kernel, user level design and how applications use kernel services. The Linux kernel can be improved by optimizing sharing and use the current advantages of recent improvement to scalability features. On the user level design, applications can be improved so that there is more focus on parallelism since some programs have not implemented those improved features. The final aspect of improving scalability is how an application uses kernel services to better share resources so that different aspects of the program are not conflicting over the same services. All of the bottlenecks are found easily and actually only take simple changes to correct or avoid.[[#Foot1|1]]

This research uses a foundation of previous research discovered during the development of scalability in UNIX systems. The major developments from shared memory machines[[#Foot2|2]] and wait-free synchronization to fast message passing ended up creating a base set of techniques, which can be used to improve scalability. These techniques have been incorporated in all major operating system including Linux, Mac OS X and Windows. Linux has been improved with kernel subsystems, such as Read-Copy-Update, which is an algorithm that is used to avoid locks and atomic instructions which affect scalability.[[#Foot3|3]] There is an excellent base of research on Linux scalability studies that have already been written, on which this research paper can model its testing standards. These papers include research on improving scalability on a 32-core machine.[[#Foot4|4]] In addition, the base of studies can be used to improve the results of these experiments by learning from the previous results. This research may also aid in identifying bottlenecks which speed up creating solutions for those problems.

==Contribution==
===16 scalability improvements===
There were 16 scalability problems encountered by MOSBENCH applications within the scope of the paper. Each was fixed. The fixes add 2617 lines of code and remove 385 lines of code from Linux.[[#Foot1|1]]

===MOSBENCH===
MOSBENCH is a set of application available through MIT. They are designed to measure scalability: "It consists of applications that previous work has shown not to scale well on Linux and applications that are designed for parallel execution and are kernel intensive."[[#Foot6|6]]

===Techniques to improve scalability===

====What hinders scalability: ''Section 4.1''====
Amdahl's Law states that a parallel program can only be sped up by the inverse of the portion of the program that cannot be made parallel, so, for example, if a program is 50% non-parallel, then at most the program can be sped up to twice the speed using parallelism. So the more serial a program is, the less capability it has for scalability. The main problems found within the MOSBENCH applications that cause serialized interactions were locking of shared data structures, writing to shared memory, competing for space in shared hardware caches, competing for shared hardware resources, and the lack of tasks leading to idle cores. These problems become more evident as more cores are added to the system. The team behind the paper came up with some solutions that either fixed these problems or avoided most, if not all, of the bottle-necking that occurred.[[#Foot1|1]]

====Multicore packet processing: ''Section 4.2''====
Linux packet processing technique requires the packets to travel along several queues before it finally becomes available for the application to use. This technique works well for most general socket applications. In recent kernel releases Linux takes advantage of multiple hardware queues (when available on the given network interface) or Receive Packet Steering[1] to direct packet flow onto different cores for processing. Or even go as far as directing packet flow to the core on which the application is running using Receive Flow Steering[2] for even better performance. Linux also attempts to increase performance using a sampling technique where it checks every 20th outgoing packet and directs flow based on its hash. This poses a problem for short lived connections like those associated with Apache since there is great potential for packets to be misdirected.

In general this technique performs poorly when there are numerous open connections spread across multiple cores due to mutex (mutual exclusion) delays and cache misses. In such scenarios its better to process all connections, with associated packets and queues, on one core to avoid said issues. The patched kernel's implementation proposed in this article uses multiple hardware queues (which can be accomplished through Receive Packet Sharing) to direct all packets from a given connection to the same core. In turn Apache is modified to only accept connections if the thread dedicated to processing them is on the same core. If the current core's queue is found to be empty it will attempt to obtain work from queues located on different cores. This configuration is ideal for numerous short connections as all the work for them in accomplished quickly on one core avoiding unnecessary mutex delays associated with packet queues and inter-core cache misses.

[1] J. Corbet. Receive Packet Steering, November 2009. http://lwn.net/Articles/362339/.

[2] J. Edge. Receive Flow Steering, April 2010. http://lwn.net/Articles/382428/.

====Sloppy counters: ''Section 4.3''====
Bottlenecks were encountered when the applications undergoing testing were referencing and updating shared counters for multiple cores. The solution in the paper is to use sloppy counters, which gets each core to track its own separate counts of references and uses a central shared counter to keep all counts on track. This is ideal because each core updates its counts by modifying its per-core counter, usually only needing access to its own local cache, cutting down on waiting for locks or serialization. Sloppy counters are also backwards-compatible with existing shared-counter code, making its implementation much easier to accomplish. The main disadvantages of the sloppy counters are that in situations where object de-allocation occurs often, because the de-allocation itself is an expensive operation, and the counters use up space proportional to the number of cores.[[#Foot1|1]]

====Lock-free comparison & Avoiding unnecessary locking: ''Section 4.4 & 4.7''====
The traditional Linux kernel has very low scalability for name lookups in the directory entry cache. This means there is reduced performance in returning information pertaining to a specific file path when there are multiple threads trying to access files in common parent directories due to the kernel serializing the process. This problem is solved in the patched kernel by introducing a new counter to keep track of threads actively looking at the directory entry cache. If a certain thread threatens an entry currently in use by another, the default locking protocol is used to avoid race conditions. If the activities have no bearing on each other the situation is rightfully ignored allowing for much faster access to different different entries in the directory entry cache.[[#Foot1|1]]

There are many other locks/mutexes that have special cases where they don't need to lock. Others can be split from locking the whole data structure to locking a part of it. Both these changes remove or reduce bottlenecks.

====Per-Core Data Structures: ''Section 4.5''====
Three centralized data structures were causing bottlenecks due to lock contention - a per-superblock list of open files, vfsmount table, the packet buffers free list. Each data structure was decentralized to per-core versions of itself. In the case of vfsmount the central data structure was maintained, and any per-core misses got written from the central table to the per-core table.

====Eliminating false sharing: ''Section 4.6''====
Misplaced variables on the cache cause different cores to request the same line, but different variables. With two different cores requesting the same variable one reading, the other writing there was a severe bottleneck. By moving the often written variable to another line the bottleneck was removed.

===Conclusion===
====memcached: ''Section 5.3''====
memcached nearly doubles throughput at 48 cores on the paper's implementation over the stock implementation. After improvements to the Linux kernel memcached is limited by hardware. Improvements to hardware scalability, virtual queue handling in this case, will allow further improvements to memcached.

====Apache: ''Section 5.4''====
Apache keeps fairly a equal amount of throughput up to 36 cores. Then it slopes downwards. At 48 cores it still has an improved throughput of more than 12 times. Apache like memcached is limited by hardware. At higher core count the network card simply cannot handle the number of packets, and the FIFO queue it holds for them overflows.

====gmake: ''Section 5.6''====
gmakes run time is nearly unchanged by the implementation changes presented in this paper. This is largely because the program has serial sections of code and some processes that finish somewhat later then all others, which prevents perfect scalability. Even then it achieves the greatest level of scalability out of the three programs (35X speed up for 48 cores.)

====No reason to give up traditional kernel design====
The contribution of this paper is a lot of research that has focus upon techniques and methods for scalability. This is accomplished through programming of applications alongside kernel programming. This research contributes by evaluating the scalability discrepancies of applications programming and kernel programming. Key discoveries in this research show the effectiveness of the kernel in handling scaling amongst CPU cores. In looking at the issue of scalability it is important to note the factors which hinder scalability.

It has been shown that simple scaling techniques can be effective in increasing scalability. The authors looked at three different approaches to removing the bottlenecks within the system. The first was to see if there were issues within the linux kernel application, the second was to identify issues with the application design and the third was to address how the application interacts with the linux kernel services. Through this approach, the authors were able to quickly identify problems such as bottlenecks and apply simple techniques in fixing the issues at hand to reap some beneficial aspects. Some of the sections listed provide insight into the improvements that can be reaped from these optimizations.

Through this research on various techniques as listed above, it was determined by the authors that the Linux kernel itself has many incorporated techniques used to improve scalability. The authors actually go on to speculate that "perhaps it is the case that Linux's system-call API is well suited to an implementation that avoids unnecessary contention over kernel objects." This tends to show that the work in the Linux community has improved Linux a large amount and is current with modern techniques for optimization.

It could also be interpreted from the paper that it may be to the benefit of the community to change how the applications are programmed rather than make changes to the Linux kernel in order to make scalability improvements. This may indicate that what has come before is done quite well when considering that the Linux kernel optimizations showed more improvement when put in conjunction with the application improvements.

Thus, Linux is only as scalable as its applications. If the kernel design is a limiting factor there is no indication of it. At the end of the experiment all limits on scalability were either application side or in IO.

==Critique==
Aside from a few acronyms that were unexplained (e.g. Linux TLB) the paper has no real stylistic problems.

===memcached: ''Section 5.3''===
memcached is treated with near perfect fairness in the paper. Its an in-memory service, so the ignored storage IO bottleneck does not affect it at all. Likewise the "stock" and "PK" implementations are given the same test suite, so there is no advantage given to either. memcached itself is non-scalable, so the MIT team was forced to run one instance per-core to keep up throughput. The FAQ at memcached.org's wiki suggests using multiple implementations per-server as a work around to another problem, which implies that there is no great problem with running multiple servers on one machine [3].

===Apache: ''Section 5.4''===
Linux has a built in kernel flaw where network packets are forced to travel though multiple queues before they arrive at queue where they can be processed by the application. This imposes significant costs on multi-core systems due to queue locking costs. This flaw inherently diminishes the performance of Apache on multi-core system due to multiple threads spread across cores being forced to deal with these mutex (mutual exclusion) costs. For the sake of this experiment Apache had a separate instance on every core listening on different ports which is not a practical real world application but merely an attempt to implement better parallel execution on a traditional kernel. The patched kernel implementation of the network stack is also specific to the problem at hand, which is processing multiple short lived connections across multiple cores. Although this provides a performance increase in the given scenario, in more general applications network performance might suffer. These tests were also rigged to avoid bottlenecks in place by network and file storage hardware. Meaning, making the proposed modifications to the kernel wont necessarily produce the same increase in productivity as described in the article. This is very much evident in the test where performance degrades past 36 cores due to limitation of the networking hardware.

===gmake: ''Section 5.6''===
Since the inherent nature of gmake makes it quite parallel, the testing and updating that were attempted on gmake resulted in essentially the same scalability results for both the stock and modified kernel. The only change that was found was that gmake spent slightly less time at the system level because of the changes that were made to the system's caching. As stated in the paper, the execution time of gmake relies quite heavily on the compiler that is uses with gmake, so depending on which compiler was chosen, gmake could run worse or even slightly better. In any case, there seems to be no fairness concerns when it comes to the scalability testing of gmake as the same application load-out was used for all of the tests.

===Conclusion: ''Sections 6 & 7''===
There is no faulty logic in the conclusion, no out right lies, no fallacies. Thus its fair. I do not know however if its valid. -[[Rannath]]

"It could also be interpreted from the paper that it may be to the benefit of the community to change how the applications are programmed rather than make changes to the Linux kernel in order to make scalability improvements."

I think that needs to be rephrased. In certain cases changes were needed to be made at kernel level to improve performance along side application code. I think the point of the article was that no changes needed to be made to the kernel design but the kernel itself obviously needs work. -kirill

==References==

[3] memcached's wiki: http://code.google.com/p/memcached/wiki/FAQ#Can_I_use_different_size_caches_across_servers_and_will_memcache

[7] MOSBENCH: http://pdos.csail.mit.edu/mosbench/

You will almost certainly have to refer to other resources; please cite these resources in the style of citation of the papers assigned (inlined numbered references). Place your bibliographic entries in this section.

'''the paper itself doesn't need to be referenced more than once as this is a critique of the paper...'''
[1] Silas Boyd-Wickizer et al. "An Analysis of Linux Scalability to Many Cores". In ''OSDI '10, 9th USENIX Symposium on OS Design and Implementation'', Vancouver, BC, Canada, 2010. http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf.

gmake:

[http://www.gnu.org/software/make/manual/make.html gmake Manual]

[http://www.gnu.org/software/make/ gmake Main Page]

==Deprecated==
===Background Concepts===
* Exim: ''Section 3.1'':
**Exim is a mail server for Unix. It's fairly parallel. The server forks a new process for each connection and twice to deliver each message. It spends 69% of its time in the kernel on a single core.

* PostgreSQL: ''Section 3.4'':
**As implied by the name PostgreSQL is a SQL database. PostgreSQL starts a separate process for each connection and uses kernel locking interfaces extensively to provide concurrent access to the database. Due to bottlenecks introduced in its code and in the kernel code, the amount of time PostgreSQL spends in the kernel increases very rapidly with addition of new cores. On a single core system PostgreSQL spends only 1.5% of its time in the kernel. On a 48 core system the execution time in the kernel jumps to 82%.

Talk:COMP 3000 Essay 2 2010 Question 1

2010-12-03T03:16:29Z

Kkashigi: /* To Do */

=Class and Notices=
(Nov. 30, 2010) Prof. Anil stated that we should focus on the 3 easiest to understand parts in section 5 and elaborate on them.

- Also, I, Daniel B., work Thursday night, so I will be finishing up as much of my part as I can for the essay before Thursday morning's class, then maybe we can all meet up in a lab in HP and put the finishing touches on the essay. I will be available online Wednesday night from about 6:30pm onwards and will be in the game dev lab or CCSS lounge Wednesday morning from about 11am to 2pm if anyone would like to meet up with me at those times.

- [[I suggest we meet up Thursday morning after Operating Systems in order to discuss and finalize the essay. Maybe we can even designate a lab for the group to meet up in. Any suggestions?]] - Daniel B.

- HP 3115 since there wont be a class in there (as its our tutorial and we know there won't be anyone there)
-- Go to Wireless Lab next to CCSS Lounge. Andrew and Dan B. will be there.

- If its all the same to you guys mind if I just join you via msn or iirc? Or phone if you really want. -Rannath

- I'm working today, but I'll be at a computer reading this page/contributing to my section. Depending on how busy I am, I should be able to get some significant writing in before 4pm today on my section and any additional sections required. RP

- I wont be there either. that does not mean i wont/cant contribute. I'll be on msn or you can just email me. -kirill

=Group members=
Patrick Young [mailto:Rannath@gmail.com Rannath@gmail.com]

Daniel Beimers [mailto:demongyro@gmail.com demongyro@gmail.com]

Andrew Bown [mailto:abown2@connect.carleton.ca abown2@connect.carleton.ca]

Kirill Kashigin [mailto:k.kashigin@gmail.com k.kashigin@gmail.com]

Rovic Perdon [mailto:rperdon@gmail.com rperdon@gmail.com]

Daniel Sont [mailto:dan.sont@gmail.com dan.sont@gmail.com]

=Methodology=
We should probably have our work verified by at least one group member before posting to the actual page

=To Do=
*contribution (conclusion mostly): just need to re-work some sections and follow the cues I left in the Conclusion section.
*critique (conclusion mostly): critique the conclusion of the essay
*style: the style section is largely untouched. Daniel and I([[Rannath]]) have puts some thoughts there, but that section needs to be made into sentences.
*yea i didn't bother rereading for changes my b, just trying to make sure our information is valid ya kno? on that note what are we looking at to get thing this on the final page?

=Essay=
==Paper - DONE!!!==
This paper was authored by - Silas Boyd-Wickizer, Austin T. Clements, Yandong Mao, Aleksey Pesterev, M. Frans Kaashoek, Robert Morris, and Nickolai Zeldovich.

They all work at MIT CSAIL.

The paper: [http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf An Analysis of Linux Scalability to Many Cores]

==Background Concepts - DONE!!!==
===memcached: ''Section 3.2''===
memcached is an in-memory hash table server. One instance of memcached running on many different cores is bottlenecked by an internal lock, which is avoided by the MIT team by running one instance per core. Clients each connect to a single instance of memcached, allowing the server to simulate parallelism without needing to make major changes to the application or kernel. With few requests, memcached spends 80% of its time in the kernel on one core, mostly processing packets.[1]

===Apache: ''Section 3.3''===
Apache is a web server that has been used in previous Linux scalability studies. In the case of this study, Apache has been configured to run a separate process on each core. Each process, in turn, has multiple threads (making it a perfect example of parallel programming). Each process uses one of their threads to accepting incoming connections and others are used to process these connections. On a single core processor, Apache spends 60% of its execution time in the kernel.[1]

===gmake: ''Section 3.5''===
gmake is an unofficial default benchmark in the Linux community which is used in this paper to build the Linux kernel. gmake takes a file called a makefile and processes its recipes for the requisite files to determine how and when to remake or recompile code. With a simple command -j or --jobs, gmake can process many of these recipes in parallel. Since gmake creates more processes than cores, it can make proper use of multiple cores to process the recipes.[2] Since gmake involves much reading and writing, in order to prevent bottlenecks due to the filesystem or hardware, the test cases use an in-memory filesystem tmpfs, which gives them a backdoor around the bottlenecks for testing purposes. In addition to this, gmake is limited in scalability due to the serial processes that run at the beginning and end of its execution, which limits its scalability to a small degree. gmake spends much of its execution time with its compiler, processing the recipes and recompiling code, but still spend 7.6% of its time in system time.[1]

[2] http://www.gnu.org/software/make/manual/make.html

==Research problem - DONE!!!==
As technology progresses, the number of core a main processor can have is increasing at an impressive rate. Soon personal computers will have so many cores that scalability will be an issue. There has to be a way that standard user level Linux kernel will scale with a 48-core system[[#Foot1|1]]. The problem with a standard Linux OS is that they are not designed for massive scalability, which will soon prove to be a problem. The issue with scalability is that a solo core will perform much more work compared to a single core working with 47 other cores. Although traditional logic states that the situation makes sense because there are 48 cores dividing the work, the information should be processed as fast as possible with each core doing as much work as possible.

To fix those scalability issues, it is necessary to focus on three major areas: the Linux kernel, user level design and how applications use kernel services. The Linux kernel can be improved by optimizing sharing and use the current advantages of recent improvement to scalability features. On the user level design, applications can be improved so that there is more focus on parallelism since some programs have not implemented those improved features. The final aspect of improving scalability is how an application uses kernel services to better share resources so that different aspects of the program are not conflicting over the same services. All of the bottlenecks are found easily and actually only take simple changes to correct or avoid.[[#Foot1|1]]

This research uses a foundation of previous research discovered during the development of scalability in UNIX systems. The major developments from shared memory machines[[#Foot2|2]] and wait-free synchronization to fast message passing ended up creating a base set of techniques, which can be used to improve scalability. These techniques have been incorporated in all major operating system including Linux, Mac OS X and Windows. Linux has been improved with kernel subsystems, such as Read-Copy-Update, which is an algorithm that is used to avoid locks and atomic instructions which affect scalability.[[#Foot3|3]] There is an excellent base of research on Linux scalability studies that have already been written, on which this research paper can model its testing standards. These papers include research on improving scalability on a 32-core machine.[[#Foot4|4]] In addition, the base of studies can be used to improve the results of these experiments by learning from the previous results. This research may also aid in identifying bottlenecks which speed up creating solutions for those problems.

==Contribution==
===16 scalability improvements===
There were 16 scalability problems encountered by MOSBENCH applications. Each was fixed. The fixes add 2617 lines of code and remove 385 lines of code from Linux.

===MOSBENCH===
MOSBENCH is a set of application available through MIT. They are designed to measure scalability: "It consists of applications that previous work has shown not to scale well on Linux and applications that are designed for parallel execution and are kernel intensive."[7]

===Techniques to improve scalability===

====What hinders scalability: ''Section 4.1''====
*The percentage of serialization in a program has a lot to do with how much an application can be sped up. This is Amdahl's Law
** Amdahl's Law states that a parallel program can only be sped up by the inverse of the proportion of the program that cannot be made parallel (e.g. 25%(.25) non-parallel --> limit of 4x speedup) (I can't get this to sound right someone fix it please -[[Rannath]] <- I will fix [[Daniel B.]]
*Types of serializing interactions found in the MOSBENCH apps:
**Locking of shared data structure as the number of cores increase leads to an increase in lock wait time
**Writing to shared memory as the number of cores increase leads to an increase in the execution time of the cache coherence protocol
**Competing for space in shared hardware cache as the number of cores increase leads to an increase in cache miss rate
**Competing for shared hardware resources as the number of cores increase leads to time lost waiting for resources
**Not enough tasks for cores leads to idle cores

====Multicore packet processing: ''Section 4.2''====
Linux packet processing technique requires the packets to travel along several queues before it finally becomes available for the application to use. This technique works well for most general socket applications. In recent kernel releases Linux takes advantage of multiple hardware queues (when available on the given network interface) or Receive Packet Steering[1] to direct packet flow onto different cores for processing. Or even go as far as directing packet flow to the core on which the application is running using Receive Flow Steering[2] for even better performance. Linux also attempts to increase performance using a sampling technique where it checks every 20th outgoing packet and directs flow based on its hash. This poses a problem for short lived connections like those associated with Apache since there is great potential for packets to be misdirected.

In general this technique performs poorly when there are numerous open connections spread across multiple cores due to mutex (mutual exclusion) delays and cache misses. In such scenarios its better to process all connections, with associated packets and queues, on one core to avoid said issues. The patched kernel's implementation proposed in this article uses multiple hardware queues (which can be accomplished through Receive Packet Sharing) to direct all packets from a given connection to the same core. In turn Apache is modified to only accept connections if the thread dedicated to processing them is on the same core. If the current core's queue is found to be empty it will attempt to obtain work from queues located on different cores. This configuration is ideal for numerous short connections as all the work for them in accomplished quickly on one core avoiding unnecessary mutex delays associated with packet queues and inter-core cache misses.

[1] J. Corbet. Receive Packet Steering, November 2009. http://lwn.net/Articles/362339/.

[2] J. Edge. Receive Flow Steering, April 2010. http://lwn.net/Articles/382428/.

====Sloppy counters: ''Section 4.3''====
Bottlenecks were encountered when the applications undergoing testing were referencing and updating shared counters for multiple cores. The solution in the paper is to use sloppy counters, which gets each core to track its own separate counts of references and uses a central shared counter to keep all counts on track. This is ideal because each core updates its counts by modifying its per-core counter, usually only needing access to its own local cache, cutting down on waiting for locks or serialization. Sloppy counters are also backwards-compatible with existing shared-counter code, making its implementation much easier to accomplish. The main disadvantages of the sloppy counters are that in situations where object de-allocation occurs often, because the de-allocation itself is an expensive operation, and the counters use up space proportional to the number of cores.

====Lock-free comparison & Avoiding unnecessary locking: ''Section 4.4 & 4.7''====
The traditional Linux kernel has very low scalability for name lookups in the directory entry cache. This means there is reduced performance in returning information pertaining to a specific file path when there are multiple threads trying to access files in common parent directories due to the kernel serializing the process. This problem is solved in the patched kernel by introducing a new counter to keep track of threads actively looking at the directory entry cache. If a certain thread threatens an entry currently in use by another, the default locking protocol is used to avoid race conditions. If the activities have no bearing on each other the situation is rightfully ignored allowing for much faster access to different different entries in the directory entry cache.

There are many other locks/mutexes that have special cases where they don't need to lock. Others can be split from locking the whole data structure to locking a part of it. Both these changes remove or reduce bottlenecks.

====Per-Core Data Structures: ''Section 4.5''====
Three centralized data structures were causing bottlenecks due to lock contention - a per-superblock list of open files, vfsmount table, the packet buffers free list. Each data structure was decentralized to per-core versions of itself. In the case of vfsmount the central data structure was maintained, and any per-core misses got written from the central table to the per-core table.

====Eliminating false sharing: ''Section 4.6''====
Misplaced variables on the cache cause different cores to request the same line, but different variables. With two different cores requesting the same variable one reading, the other writing there was a severe bottleneck. By moving the often written variable to another line the bottleneck was removed.

===Conclusion===
====memcached: ''Section 5.3''====
memcached nearly doubles throughput at 48 cores on the paper's implementation over the stock implementation. After improvements to the Linux kernel memcached is limited by hardware. Improvements to hardware scalability, virtual queue handling in this case, will allow further improvements to memcached.

====Apache: ''Section 5.4''====
Apache keeps fairly a equal amount of throughput up to 36 cores. Then it slopes downwards. At 48 cores it still has an improved throughput of more than 12 times. Apache like memcached is limited by hardware. At higher core count the network card simply cannot handle the number of packets, and the FIFO queue it holds for them overflows.

====gmake: ''Section 5.6''====
gmakes run time is nearly unchanged by the implementation changes presented in this paper. This is largely because the program has serial sections of code and some processes that finish somewhat later then all others, which prevents perfect scalability. Even then it achieves the greatest level of scalability out of the three programs (35X speed up for 48 cores.)

====No reason to give up traditional kernel design====

The contribution of this paper is a lot of research that has focus upon techniques and methods for scalability. This is accomplished through programming of applications alongside kernel programming. This research contributes by evaluating the scalability discrepancies of applications programming and kernel programming. Key discoveries in this research show the effectiveness of the kernel in handling scaling amongst CPU cores. In looking at the issue of scalability it is important to note the causes of the factors which hinder scalability.

It has been shown that simple scaling techniques can be effective in increasing scalability. The authors looked at three different approaches to removing the bottlenecks within the system. The first was to see if there were issues within the linux kernel application, the second was to identify issues with the application design and the third was to address how the application interacts with the linux kernel services. Through this approach, the authors were able to quickly identify problems such as bottlenecks and apply simple techniques in fixing the issues at hand to reap some beneficial aspects. Some of the sections listed provide insight into the improvements that can be reaped from these optimizations.

Through this research on various techniques as listed above, it was determined by the authors that the Linux kernel itself has many incorporated techniques used to improve scalability. The authors actually go on to speculate that "perhaps it is the case that Linux's system-call API is well suited to an implementation that avoids unnecessary contention over kernel objects." This tends to show that the work in the Linux community has improved Linux a large amount and is current with modern techniques for optimization. It could also be interpreted from the paper that it may be to the benefit of the community to change how the applications are programmed rather than make changes to the Linux kernel in order to make scalability improvements. This may indicate that what has come before is done quite well when considering that the Linux kernel optimizations showed more improvement when put in conjunction with the application improvements.

The hypothesis of the paper was the viability of traditional kernel design. This should explicitly tie into that.
Also the fact that more then half of the programs were limited by IO should be mentioned.

==Critique==
Aside from a few acronyms that were unexplained (e.g. Linux TLB) the paper has no real stylistic problems.

===memcached: ''Section 5.3''===
memcached is treated with near perfect fairness in the paper. Its an in-memory service, so the ignored storage IO bottleneck does not affect it at all. Likewise the "stock" and "PK" implementations are given the same test suite, so there is no advantage given to either. memcached itself is non-scalable, so the MIT team was forced to run one instance per-core to keep up throughput. The FAQ at memcached.org's wiki suggests using multiple implementations per-server as a work around to another problem, which implies that there is no great problem with running multiple servers on one machine [3].

===Apache: ''Section 5.4''===
Linux has a built in kernel flaw where network packets are forced to travel though multiple queues before they arrive at queue where they can be processed by the application. This imposes significant costs on multi-core systems due to queue locking costs. This flaw inherently diminishes the performance of Apache on multi-core system due to multiple threads spread across cores being forced to deal with these mutex (mutual exclusion) costs. For the sake of this experiment Apache had a separate instance on every core listening on different ports which is not a practical real world application but merely an attempt to implement better parallel execution on a traditional kernel. The patched kernel implementation of the network stack is also specific to the problem at hand, which is processing multiple short lived connections across multiple cores. Although this provides a performance increase in the given scenario, in more general applications network performance might suffer. These tests were also rigged to avoid bottlenecks in place by network and file storage hardware. Meaning, making the proposed modifications to the kernel wont necessarily produce the same increase in productivity as described in the article. This is very much evident in the test where performance degrades past 36 cores due to limitation of the networking hardware.

===gmake: ''Section 5.6''===
Since the inherent nature of gmake makes it quite parallel, the testing and updating that were attempted on gmake resulted in essentially the same scalability results for both the stock and modified kernel. The only change that was found was that gmake spent slightly less time at the system level because of the changes that were made to the system's caching. As stated in the paper, the execution time of gmake relies quite heavily on the compiler that is uses with gmake, so depending on which compiler was chosen, gmake could run worse or even slightly better. In any case, there seems to be no fairness concerns when it comes to the scalability testing of gmake as the same application load-out was used for all of the tests.

===Conclusion: ''Sections 6 & 7''===
Given that all tests are more or less fair for the purposes of the benchmarks. They would support the Hypothesis that Linux can be made to scale, at least to 48 cores. Thus the conclusion is fair iff the rest of the paper is fair.

Now you just have to fill in how fair the rest of the paper is.

==References==

[3] memcached's wiki: http://code.google.com/p/memcached/wiki/FAQ#Can_I_use_different_size_caches_across_servers_and_will_memcache

[7] MOSBENCH: http://pdos.csail.mit.edu/mosbench/

You will almost certainly have to refer to other resources; please cite these resources in the style of citation of the papers assigned (inlined numbered references). Place your bibliographic entries in this section.

'''the paper itself doesn't need to be referenced more than once as this is a critique of the paper...'''
[1] Silas Boyd-Wickizer et al. "An Analysis of Linux Scalability to Many Cores". In ''OSDI '10, 9th USENIX Symposium on OS Design and Implementation'', Vancouver, BC, Canada, 2010. http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf.

gmake:

[http://www.gnu.org/software/make/manual/make.html gmake Manual]

[http://www.gnu.org/software/make/ gmake Main Page]

==Deprecated==
===Background Concepts===
* Exim: ''Section 3.1'':
**Exim is a mail server for Unix. It's fairly parallel. The server forks a new process for each connection and twice to deliver each message. It spends 69% of its time in the kernel on a single core.

* PostgreSQL: ''Section 3.4'':
**As implied by the name PostgreSQL is a SQL database. PostgreSQL starts a separate process for each connection and uses kernel locking interfaces extensively to provide concurrent access to the database. Due to bottlenecks introduced in its code and in the kernel code, the amount of time PostgreSQL spends in the kernel increases very rapidly with addition of new cores. On a single core system PostgreSQL spends only 1.5% of its time in the kernel. On a 48 core system the execution time in the kernel jumps to 82%.

Talk:COMP 3000 Essay 2 2010 Question 1

2010-12-03T03:14:29Z

Kkashigi: /* Apache: Section 5.4 */

=Class and Notices=
(Nov. 30, 2010) Prof. Anil stated that we should focus on the 3 easiest to understand parts in section 5 and elaborate on them.

- Also, I, Daniel B., work Thursday night, so I will be finishing up as much of my part as I can for the essay before Thursday morning's class, then maybe we can all meet up in a lab in HP and put the finishing touches on the essay. I will be available online Wednesday night from about 6:30pm onwards and will be in the game dev lab or CCSS lounge Wednesday morning from about 11am to 2pm if anyone would like to meet up with me at those times.

- [[I suggest we meet up Thursday morning after Operating Systems in order to discuss and finalize the essay. Maybe we can even designate a lab for the group to meet up in. Any suggestions?]] - Daniel B.

- HP 3115 since there wont be a class in there (as its our tutorial and we know there won't be anyone there)
-- Go to Wireless Lab next to CCSS Lounge. Andrew and Dan B. will be there.

- If its all the same to you guys mind if I just join you via msn or iirc? Or phone if you really want. -Rannath

- I'm working today, but I'll be at a computer reading this page/contributing to my section. Depending on how busy I am, I should be able to get some significant writing in before 4pm today on my section and any additional sections required. RP

- I wont be there either. that does not mean i wont/cant contribute. I'll be on msn or you can just email me. -kirill

=Group members=
Patrick Young [mailto:Rannath@gmail.com Rannath@gmail.com]

Daniel Beimers [mailto:demongyro@gmail.com demongyro@gmail.com]

Andrew Bown [mailto:abown2@connect.carleton.ca abown2@connect.carleton.ca]

Kirill Kashigin [mailto:k.kashigin@gmail.com k.kashigin@gmail.com]

Rovic Perdon [mailto:rperdon@gmail.com rperdon@gmail.com]

Daniel Sont [mailto:dan.sont@gmail.com dan.sont@gmail.com]

=Methodology=
We should probably have our work verified by at least one group member before posting to the actual page

=To Do=
*contribution (conclusion mostly): just need to re-work some sections and follow the cues I left in the Conclusion section.
*critique (conclusion mostly): critique the conclusion of the essay
*style: the style section is largely untouched. Daniel and I([[Rannath]]) have puts some thoughts there, but that section needs to be made into sentences.

=Essay=
==Paper - DONE!!!==
This paper was authored by - Silas Boyd-Wickizer, Austin T. Clements, Yandong Mao, Aleksey Pesterev, M. Frans Kaashoek, Robert Morris, and Nickolai Zeldovich.

They all work at MIT CSAIL.

The paper: [http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf An Analysis of Linux Scalability to Many Cores]

==Background Concepts - DONE!!!==
===memcached: ''Section 3.2''===
memcached is an in-memory hash table server. One instance of memcached running on many different cores is bottlenecked by an internal lock, which is avoided by the MIT team by running one instance per core. Clients each connect to a single instance of memcached, allowing the server to simulate parallelism without needing to make major changes to the application or kernel. With few requests, memcached spends 80% of its time in the kernel on one core, mostly processing packets.[1]

===Apache: ''Section 3.3''===
Apache is a web server that has been used in previous Linux scalability studies. In the case of this study, Apache has been configured to run a separate process on each core. Each process, in turn, has multiple threads (making it a perfect example of parallel programming). Each process uses one of their threads to accepting incoming connections and others are used to process these connections. On a single core processor, Apache spends 60% of its execution time in the kernel.[1]

===gmake: ''Section 3.5''===
gmake is an unofficial default benchmark in the Linux community which is used in this paper to build the Linux kernel. gmake takes a file called a makefile and processes its recipes for the requisite files to determine how and when to remake or recompile code. With a simple command -j or --jobs, gmake can process many of these recipes in parallel. Since gmake creates more processes than cores, it can make proper use of multiple cores to process the recipes.[2] Since gmake involves much reading and writing, in order to prevent bottlenecks due to the filesystem or hardware, the test cases use an in-memory filesystem tmpfs, which gives them a backdoor around the bottlenecks for testing purposes. In addition to this, gmake is limited in scalability due to the serial processes that run at the beginning and end of its execution, which limits its scalability to a small degree. gmake spends much of its execution time with its compiler, processing the recipes and recompiling code, but still spend 7.6% of its time in system time.[1]

[2] http://www.gnu.org/software/make/manual/make.html

==Research problem - DONE!!!==
As technology progresses, the number of core a main processor can have is increasing at an impressive rate. Soon personal computers will have so many cores that scalability will be an issue. There has to be a way that standard user level Linux kernel will scale with a 48-core system[[#Foot1|1]]. The problem with a standard Linux OS is that they are not designed for massive scalability, which will soon prove to be a problem. The issue with scalability is that a solo core will perform much more work compared to a single core working with 47 other cores. Although traditional logic states that the situation makes sense because there are 48 cores dividing the work, the information should be processed as fast as possible with each core doing as much work as possible.

To fix those scalability issues, it is necessary to focus on three major areas: the Linux kernel, user level design and how applications use kernel services. The Linux kernel can be improved by optimizing sharing and use the current advantages of recent improvement to scalability features. On the user level design, applications can be improved so that there is more focus on parallelism since some programs have not implemented those improved features. The final aspect of improving scalability is how an application uses kernel services to better share resources so that different aspects of the program are not conflicting over the same services. All of the bottlenecks are found easily and actually only take simple changes to correct or avoid.[[#Foot1|1]]

This research uses a foundation of previous research discovered during the development of scalability in UNIX systems. The major developments from shared memory machines[[#Foot2|2]] and wait-free synchronization to fast message passing ended up creating a base set of techniques, which can be used to improve scalability. These techniques have been incorporated in all major operating system including Linux, Mac OS X and Windows. Linux has been improved with kernel subsystems, such as Read-Copy-Update, which is an algorithm that is used to avoid locks and atomic instructions which affect scalability.[[#Foot3|3]] There is an excellent base of research on Linux scalability studies that have already been written, on which this research paper can model its testing standards. These papers include research on improving scalability on a 32-core machine.[[#Foot4|4]] In addition, the base of studies can be used to improve the results of these experiments by learning from the previous results. This research may also aid in identifying bottlenecks which speed up creating solutions for those problems.

==Contribution==
===16 scalability improvements===
There were 16 scalability problems encountered by MOSBENCH applications. Each was fixed. The fixes add 2617 lines of code and remove 385 lines of code from Linux.

===MOSBENCH===
MOSBENCH is a set of application available through MIT. They are designed to measure scalability: "It consists of applications that previous work has shown not to scale well on Linux and applications that are designed for parallel execution and are kernel intensive."[7]

===Techniques to improve scalability===

====What hinders scalability: ''Section 4.1''====
*The percentage of serialization in a program has a lot to do with how much an application can be sped up. This is Amdahl's Law
** Amdahl's Law states that a parallel program can only be sped up by the inverse of the proportion of the program that cannot be made parallel (e.g. 25%(.25) non-parallel --> limit of 4x speedup) (I can't get this to sound right someone fix it please -[[Rannath]] <- I will fix [[Daniel B.]]
*Types of serializing interactions found in the MOSBENCH apps:
**Locking of shared data structure as the number of cores increase leads to an increase in lock wait time
**Writing to shared memory as the number of cores increase leads to an increase in the execution time of the cache coherence protocol
**Competing for space in shared hardware cache as the number of cores increase leads to an increase in cache miss rate
**Competing for shared hardware resources as the number of cores increase leads to time lost waiting for resources
**Not enough tasks for cores leads to idle cores

====Multicore packet processing: ''Section 4.2''====
Linux packet processing technique requires the packets to travel along several queues before it finally becomes available for the application to use. This technique works well for most general socket applications. In recent kernel releases Linux takes advantage of multiple hardware queues (when available on the given network interface) or Receive Packet Steering[1] to direct packet flow onto different cores for processing. Or even go as far as directing packet flow to the core on which the application is running using Receive Flow Steering[2] for even better performance. Linux also attempts to increase performance using a sampling technique where it checks every 20th outgoing packet and directs flow based on its hash. This poses a problem for short lived connections like those associated with Apache since there is great potential for packets to be misdirected.

In general this technique performs poorly when there are numerous open connections spread across multiple cores due to mutex (mutual exclusion) delays and cache misses. In such scenarios its better to process all connections, with associated packets and queues, on one core to avoid said issues. The patched kernel's implementation proposed in this article uses multiple hardware queues (which can be accomplished through Receive Packet Sharing) to direct all packets from a given connection to the same core. In turn Apache is modified to only accept connections if the thread dedicated to processing them is on the same core. If the current core's queue is found to be empty it will attempt to obtain work from queues located on different cores. This configuration is ideal for numerous short connections as all the work for them in accomplished quickly on one core avoiding unnecessary mutex delays associated with packet queues and inter-core cache misses.

[1] J. Corbet. Receive Packet Steering, November 2009. http://lwn.net/Articles/362339/.

[2] J. Edge. Receive Flow Steering, April 2010. http://lwn.net/Articles/382428/.

====Sloppy counters: ''Section 4.3''====
Bottlenecks were encountered when the applications undergoing testing were referencing and updating shared counters for multiple cores. The solution in the paper is to use sloppy counters, which gets each core to track its own separate counts of references and uses a central shared counter to keep all counts on track. This is ideal because each core updates its counts by modifying its per-core counter, usually only needing access to its own local cache, cutting down on waiting for locks or serialization. Sloppy counters are also backwards-compatible with existing shared-counter code, making its implementation much easier to accomplish. The main disadvantages of the sloppy counters are that in situations where object de-allocation occurs often, because the de-allocation itself is an expensive operation, and the counters use up space proportional to the number of cores.

====Lock-free comparison & Avoiding unnecessary locking: ''Section 4.4 & 4.7''====
The traditional Linux kernel has very low scalability for name lookups in the directory entry cache. This means there is reduced performance in returning information pertaining to a specific file path when there are multiple threads trying to access files in common parent directories due to the kernel serializing the process. This problem is solved in the patched kernel by introducing a new counter to keep track of threads actively looking at the directory entry cache. If a certain thread threatens an entry currently in use by another, the default locking protocol is used to avoid race conditions. If the activities have no bearing on each other the situation is rightfully ignored allowing for much faster access to different different entries in the directory entry cache.

There are many other locks/mutexes that have special cases where they don't need to lock. Others can be split from locking the whole data structure to locking a part of it. Both these changes remove or reduce bottlenecks.

====Per-Core Data Structures: ''Section 4.5''====
Three centralized data structures were causing bottlenecks due to lock contention - a per-superblock list of open files, vfsmount table, the packet buffers free list. Each data structure was decentralized to per-core versions of itself. In the case of vfsmount the central data structure was maintained, and any per-core misses got written from the central table to the per-core table.

====Eliminating false sharing: ''Section 4.6''====
Misplaced variables on the cache cause different cores to request the same line, but different variables. With two different cores requesting the same variable one reading, the other writing there was a severe bottleneck. By moving the often written variable to another line the bottleneck was removed.

===Conclusion===
====memcached: ''Section 5.3''====
memcached nearly doubles throughput at 48 cores on the paper's implementation over the stock implementation. After improvements to the Linux kernel memcached is limited by hardware. Improvements to hardware scalability, virtual queue handling in this case, will allow further improvements to memcached.

====Apache: ''Section 5.4''====
Apache keeps fairly a equal amount of throughput up to 36 cores. Then it slopes downwards. At 48 cores it still has an improved throughput of more than 12 times. Apache like memcached is limited by hardware. At higher core count the network card simply cannot handle the number of packets, and the FIFO queue it holds for them overflows.

====gmake: ''Section 5.6''====
gmakes run time is nearly unchanged by the implementation changes presented in this paper. This is largely because the program has serial sections of code and some processes that finish somewhat later then all others, which prevents perfect scalability. Even then it achieves the greatest level of scalability out of the three programs (35X speed up for 48 cores.)

====No reason to give up traditional kernel design====

The contribution of this paper is a lot of research that has focus upon techniques and methods for scalability. This is accomplished through programming of applications alongside kernel programming. This research contributes by evaluating the scalability discrepancies of applications programming and kernel programming. Key discoveries in this research show the effectiveness of the kernel in handling scaling amongst CPU cores. In looking at the issue of scalability it is important to note the causes of the factors which hinder scalability.

It has been shown that simple scaling techniques can be effective in increasing scalability. The authors looked at three different approaches to removing the bottlenecks within the system. The first was to see if there were issues within the linux kernel application, the second was to identify issues with the application design and the third was to address how the application interacts with the linux kernel services. Through this approach, the authors were able to quickly identify problems such as bottlenecks and apply simple techniques in fixing the issues at hand to reap some beneficial aspects. Some of the sections listed provide insight into the improvements that can be reaped from these optimizations.

Through this research on various techniques as listed above, it was determined by the authors that the Linux kernel itself has many incorporated techniques used to improve scalability. The authors actually go on to speculate that "perhaps it is the case that Linux's system-call API is well suited to an implementation that avoids unnecessary contention over kernel objects." This tends to show that the work in the Linux community has improved Linux a large amount and is current with modern techniques for optimization. It could also be interpreted from the paper that it may be to the benefit of the community to change how the applications are programmed rather than make changes to the Linux kernel in order to make scalability improvements. This may indicate that what has come before is done quite well when considering that the Linux kernel optimizations showed more improvement when put in conjunction with the application improvements.

The hypothesis of the paper was the viability of traditional kernel design. This should explicitly tie into that.
Also the fact that more then half of the programs were limited by IO should be mentioned.

==Critique==
Aside from a few acronyms that were unexplained (e.g. Linux TLB) the paper has no real stylistic problems.

===memcached: ''Section 5.3''===
memcached is treated with near perfect fairness in the paper. Its an in-memory service, so the ignored storage IO bottleneck does not affect it at all. Likewise the "stock" and "PK" implementations are given the same test suite, so there is no advantage given to either. memcached itself is non-scalable, so the MIT team was forced to run one instance per-core to keep up throughput. The FAQ at memcached.org's wiki suggests using multiple implementations per-server as a work around to another problem, which implies that there is no great problem with running multiple servers on one machine [3].

===Apache: ''Section 5.4''===
Linux has a built in kernel flaw where network packets are forced to travel though multiple queues before they arrive at queue where they can be processed by the application. This imposes significant costs on multi-core systems due to queue locking costs. This flaw inherently diminishes the performance of Apache on multi-core system due to multiple threads spread across cores being forced to deal with these mutex (mutual exclusion) costs. For the sake of this experiment Apache had a separate instance on every core listening on different ports which is not a practical real world application but merely an attempt to implement better parallel execution on a traditional kernel. The patched kernel implementation of the network stack is also specific to the problem at hand, which is processing multiple short lived connections across multiple cores. Although this provides a performance increase in the given scenario, in more general applications network performance might suffer. These tests were also rigged to avoid bottlenecks in place by network and file storage hardware. Meaning, making the proposed modifications to the kernel wont necessarily produce the same increase in productivity as described in the article. This is very much evident in the test where performance degrades past 36 cores due to limitation of the networking hardware.

===gmake: ''Section 5.6''===
Since the inherent nature of gmake makes it quite parallel, the testing and updating that were attempted on gmake resulted in essentially the same scalability results for both the stock and modified kernel. The only change that was found was that gmake spent slightly less time at the system level because of the changes that were made to the system's caching. As stated in the paper, the execution time of gmake relies quite heavily on the compiler that is uses with gmake, so depending on which compiler was chosen, gmake could run worse or even slightly better. In any case, there seems to be no fairness concerns when it comes to the scalability testing of gmake as the same application load-out was used for all of the tests.

===Conclusion: ''Sections 6 & 7''===
Given that all tests are more or less fair for the purposes of the benchmarks. They would support the Hypothesis that Linux can be made to scale, at least to 48 cores. Thus the conclusion is fair iff the rest of the paper is fair.

Now you just have to fill in how fair the rest of the paper is.

==References==

[3] memcached's wiki: http://code.google.com/p/memcached/wiki/FAQ#Can_I_use_different_size_caches_across_servers_and_will_memcache

[7] MOSBENCH: http://pdos.csail.mit.edu/mosbench/

You will almost certainly have to refer to other resources; please cite these resources in the style of citation of the papers assigned (inlined numbered references). Place your bibliographic entries in this section.

'''the paper itself doesn't need to be referenced more than once as this is a critique of the paper...'''
[1] Silas Boyd-Wickizer et al. "An Analysis of Linux Scalability to Many Cores". In ''OSDI '10, 9th USENIX Symposium on OS Design and Implementation'', Vancouver, BC, Canada, 2010. http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf.

gmake:

[http://www.gnu.org/software/make/manual/make.html gmake Manual]

[http://www.gnu.org/software/make/ gmake Main Page]

==Deprecated==
===Background Concepts===
* Exim: ''Section 3.1'':
**Exim is a mail server for Unix. It's fairly parallel. The server forks a new process for each connection and twice to deliver each message. It spends 69% of its time in the kernel on a single core.

* PostgreSQL: ''Section 3.4'':
**As implied by the name PostgreSQL is a SQL database. PostgreSQL starts a separate process for each connection and uses kernel locking interfaces extensively to provide concurrent access to the database. Due to bottlenecks introduced in its code and in the kernel code, the amount of time PostgreSQL spends in the kernel increases very rapidly with addition of new cores. On a single core system PostgreSQL spends only 1.5% of its time in the kernel. On a 48 core system the execution time in the kernel jumps to 82%.

Talk:COMP 3000 Essay 2 2010 Question 1

2010-12-03T02:53:08Z

Kkashigi: /* Apache: Section 5.4 */

=Class and Notices=
(Nov. 30, 2010) Prof. Anil stated that we should focus on the 3 easiest to understand parts in section 5 and elaborate on them.

- Also, I, Daniel B., work Thursday night, so I will be finishing up as much of my part as I can for the essay before Thursday morning's class, then maybe we can all meet up in a lab in HP and put the finishing touches on the essay. I will be available online Wednesday night from about 6:30pm onwards and will be in the game dev lab or CCSS lounge Wednesday morning from about 11am to 2pm if anyone would like to meet up with me at those times.

- [[I suggest we meet up Thursday morning after Operating Systems in order to discuss and finalize the essay. Maybe we can even designate a lab for the group to meet up in. Any suggestions?]] - Daniel B.

- HP 3115 since there wont be a class in there (as its our tutorial and we know there won't be anyone there)
-- Go to Wireless Lab next to CCSS Lounge. Andrew and Dan B. will be there.

- If its all the same to you guys mind if I just join you via msn or iirc? Or phone if you really want. -Rannath

- I'm working today, but I'll be at a computer reading this page/contributing to my section. Depending on how busy I am, I should be able to get some significant writing in before 4pm today on my section and any additional sections required. RP

- I wont be there either. that does not mean i wont/cant contribute. I'll be on msn or you can just email me. -kirill

=Group members=
Patrick Young [mailto:Rannath@gmail.com Rannath@gmail.com]

Daniel Beimers [mailto:demongyro@gmail.com demongyro@gmail.com]

Andrew Bown [mailto:abown2@connect.carleton.ca abown2@connect.carleton.ca]

Kirill Kashigin [mailto:k.kashigin@gmail.com k.kashigin@gmail.com]

Rovic Perdon [mailto:rperdon@gmail.com rperdon@gmail.com]

Daniel Sont [mailto:dan.sont@gmail.com dan.sont@gmail.com]

=Methodology=
We should probably have our work verified by at least one group member before posting to the actual page

=To Do=
*contribution (conclusion mostly): just need to re-work some sections and follow the cues I left in the Conclusion section.
*critique (conclusion mostly): critique the conclusion of the essay
*style: the style section is largely untouched. Daniel and I([[Rannath]]) have puts some thoughts there, but that section needs to be made into sentences.

=Essay=
==Paper - DONE!!!==
This paper was authored by - Silas Boyd-Wickizer, Austin T. Clements, Yandong Mao, Aleksey Pesterev, M. Frans Kaashoek, Robert Morris, and Nickolai Zeldovich.

They all work at MIT CSAIL.

The paper: [http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf An Analysis of Linux Scalability to Many Cores]

==Background Concepts - DONE!!!==
===memcached: ''Section 3.2''===
memcached is an in-memory hash table server. One instance of memcached running on many different cores is bottlenecked by an internal lock, which is avoided by the MIT team by running one instance per core. Clients each connect to a single instance of memcached, allowing the server to simulate parallelism without needing to make major changes to the application or kernel. With few requests, memcached spends 80% of its time in the kernel on one core, mostly processing packets.[1]

===Apache: ''Section 3.3''===
Apache is a web server that has been used in previous Linux scalability studies. In the case of this study, Apache has been configured to run a separate process on each core. Each process, in turn, has multiple threads (making it a perfect example of parallel programming). Each process uses one of their threads to accepting incoming connections and others are used to process these connections. On a single core processor, Apache spends 60% of its execution time in the kernel.[1]

===gmake: ''Section 3.5''===
gmake is an unofficial default benchmark in the Linux community which is used in this paper to build the Linux kernel. gmake takes a file called a makefile and processes its recipes for the requisite files to determine how and when to remake or recompile code. With a simple command -j or --jobs, gmake can process many of these recipes in parallel. Since gmake creates more processes than cores, it can make proper use of multiple cores to process the recipes.[2] Since gmake involves much reading and writing, in order to prevent bottlenecks due to the filesystem or hardware, the test cases use an in-memory filesystem tmpfs, which gives them a backdoor around the bottlenecks for testing purposes. In addition to this, gmake is limited in scalability due to the serial processes that run at the beginning and end of its execution, which limits its scalability to a small degree. gmake spends much of its execution time with its compiler, processing the recipes and recompiling code, but still spend 7.6% of its time in system time.[1]

[2] http://www.gnu.org/software/make/manual/make.html

==Research problem - DONE!!!==
As technology progresses, the number of core a main processor can have is increasing at an impressive rate. Soon personal computers will have so many cores that scalability will be an issue. There has to be a way that standard user level Linux kernel will scale with a 48-core system[[#Foot1|1]]. The problem with a standard Linux OS is that they are not designed for massive scalability, which will soon prove to be a problem. The issue with scalability is that a solo core will perform much more work compared to a single core working with 47 other cores. Although traditional logic states that the situation makes sense because there are 48 cores dividing the work, the information should be processed as fast as possible with each core doing as much work as possible.

To fix those scalability issues, it is necessary to focus on three major areas: the Linux kernel, user level design and how applications use kernel services. The Linux kernel can be improved by optimizing sharing and use the current advantages of recent improvement to scalability features. On the user level design, applications can be improved so that there is more focus on parallelism since some programs have not implemented those improved features. The final aspect of improving scalability is how an application uses kernel services to better share resources so that different aspects of the program are not conflicting over the same services. All of the bottlenecks are found easily and actually only take simple changes to correct or avoid.[[#Foot1|1]]

This research uses a foundation of previous research discovered during the development of scalability in UNIX systems. The major developments from shared memory machines[[#Foot2|2]] and wait-free synchronization to fast message passing ended up creating a base set of techniques, which can be used to improve scalability. These techniques have been incorporated in all major operating system including Linux, Mac OS X and Windows. Linux has been improved with kernel subsystems, such as Read-Copy-Update, which is an algorithm that is used to avoid locks and atomic instructions which affect scalability.[[#Foot3|3]] There is an excellent base of research on Linux scalability studies that have already been written, on which this research paper can model its testing standards. These papers include research on improving scalability on a 32-core machine.[[#Foot4|4]] In addition, the base of studies can be used to improve the results of these experiments by learning from the previous results. This research may also aid in identifying bottlenecks which speed up creating solutions for those problems.

==Contribution==
All contributions in this paper are the result of the identification and removal or marginalization of bottlenecks.

===What hinders scalability: ''Section 4.1''===
*The percentage of serialization in a program has a lot to do with how much an application can be sped up. This is Amdahl's Law
** Amdahl's Law states that a parallel program can only be sped up by the inverse of the proportion of the program that cannot be made parallel (e.g. 25%(.25) non-parallel --> limit of 4x speedup) (I can't get this to sound right someone fix it please -[[Rannath]] <- I will fix [[Daniel B.]]
*Types of serializing interactions found in the MOSBENCH apps:
**Locking of shared data structure as the number of cores increase leads to an increase in lock wait time
**Writing to shared memory as the number of cores increase leads to an increase in the execution time of the cache coherence protocol
**Competing for space in shared hardware cache as the number of cores increase leads to an increase in cache miss rate
**Competing for shared hardware resources as the number of cores increase leads to time lost waiting for resources
**Not enough tasks for cores leads to idle cores

===Multicore packet processing: ''Section 4.2''===
Linux packet processing technique requires the packets to travel along several queues before it finally becomes available for the application to use. This technique works well for most general socket applications. In recent kernel releases Linux takes advantage of multiple hardware queues (when available on the given network interface) or Receive Packet Steering[1] to direct packet flow onto different cores for processing. Or even go as far as directing packet flow to the core on which the application is running using Receive Flow Steering[2] for even better performance. Linux also attempts to increase performance using a sampling technique where it checks every 20th outgoing packet and directs flow based on its hash. This poses a problem for short lived connections like those associated with Apache since there is great potential for packets to be misdirected.

In general this technique performs poorly when there are numerous open connections spread across multiple cores due to mutex (mutual exclusion) delays and cache misses. In such scenarios its better to process all connections, with associated packets and queues, on one core to avoid said issues. The patched kernel's implementation proposed in this article uses multiple hardware queues (which can be accomplished through Receive Packet Sharing) to direct all packets from a given connection to the same core. In turn Apache is modified to only accept connections if the thread dedicated to processing them is on the same core. If the current core's queue is found to be empty it will attempt to obtain work from queues located on different cores. This configuration is ideal for numerous short connections as all the work for them in accomplished quickly on one core avoiding unnecessary mutex delays associated with packet queues and inter-core cache misses.

[1] J. Corbet. Receive Packet Steering, November 2009. http://lwn.net/Articles/362339/.

[2] J. Edge. Receive Flow Steering, April 2010. http://lwn.net/Articles/382428/.

===Sloppy counters: ''Section 4.3''===
Bottlenecks were encountered when the applications undergoing testing were referencing and updating shared counters for multiple cores. The solution in the paper is to use sloppy counters, which gets each core to track its own separate counts of references and uses a central shared counter to keep all counts on track. This is ideal because each core updates its counts by modifying its per-core counter, usually only needing access to its own local cache, cutting down on waiting for locks or serialization. Sloppy counters are also backwards-compatible with existing shared-counter code, making its implementation much easier to accomplish. The main disadvantages of the sloppy counters are that in situations where object de-allocation occurs often, because the de-allocation itself is an expensive operation, and the counters use up space proportional to the number of cores.

===Lock-free comparison & Avoiding unnecessary locking: ''Section 4.4 & 4.7''===
The traditional Linux kernel has very low scalability for name lookups in the directory entry cache. This means there is reduced performance in returning information pertaining to a specific file path when there are multiple threads trying to access files in common parent directories due to the kernel serializing the process. This problem is solved in the patched kernel by introducing a new counter to keep track of threads actively looking at the directory entry cache. If a certain thread threatens an entry currently in use by another, the default locking protocol is used to avoid race conditions. If the activities have no bearing on each other the situation is rightfully ignored allowing for much faster access to different different entries in the directory entry cache.

There are many other locks/mutexes that have special cases where they don't need to lock. Others can be split from locking the whole data structure to locking a part of it. Both these changes remove or reduce bottlenecks.

===Per-Core Data Structures: ''Section 4.5''===
Three centralized data structures were causing bottlenecks due to lock contention - a per-superblock list of open files, vfsmount table, the packet buffers free list. Each data structure was decentralized to per-core versions of itself. In the case of vfsmount the central data structure was maintained, and any per-core misses got written from the central table to the per-core table.

===Eliminating false sharing: ''Section 4.6''===
Misplaced variables on the cache cause different cores to request the same line, but different variables. With two different cores requesting the same variable one reading, the other writing there was a severe bottleneck. By moving the often written variable to another line the bottleneck was removed.

===memcached: ''Section 5.3''===
memcached nearly doubles throughput at 48 cores on the paper's implementation over the stock implementation. After improvements to the Linux kernel memcached is limited by hardware. Improvements to hardware scalability, virtual queue handling in this case, will allow further improvements to memcached.

===Apache: ''Section 5.4''===
Apache keeps fairly a equal amount of throughput up to 36 cores. Then it slopes downwards. At 48 cores it still has an improved throughput of more than 12 times. Apache like memcached is limited by hardware. At higher core count the network card simply cannot handle the number of packets, and the FIFO queue it holds for them overflows.

===gmake: ''Section 5.6''===
gmakes run time is nearly unchanged by the implementation changes presented in this paper. This is largely because the program has serial sections of code and some processes that finish somewhat later then all others, which prevents perfect scalability. Even then it achieves the greatest level of scalability out of the three programs (35X speed up for 48 cores.)

===Conclusion: ''Sections 6 & 7''===
The contribution of this paper is a lot of research that has focus upon techniques and methods for scalability. This is accomplished through programming of applications alongside kernel programming. This research contributes by evaluating the scalability discrepancies of applications programming and kernel programming. Key discoveries in this research show the effectiveness of the kernel in handling scaling amongst CPU cores. In looking at the issue of scalability it is important to note the causes of the factors which hinder scalability.

It has been shown that simple scaling techniques can be effective in increasing scalability. The authors looked at three different approaches to removing the bottlenecks within the system. The first was to see if there were issues within the linux kernel application, the second was to identify issues with the application design and the third was to address how the application interacts with the linux kernel services. Through this approach, the authors were able to quickly identify problems such as bottlenecks and apply simple techniques in fixing the issues at hand to reap some beneficial aspects. Some of the sections listed provide insight into the improvements that can be reaped from these optimizations.

Through this research on various techniques as listed above, it was determined by the authors that the Linux kernel itself has many incorporated techniques used to improve scalability. The authors actually go on to speculate that "perhaps it is the case that Linux's system-call API is well suited to an implementation that avoids unnecessary contention over kernel objects." This tends to show that the work in the Linux community has improved Linux a large amount and is current with modern techniques for optimization. It could also be interpreted from the paper that it may be to the benefit of the community to change how the applications are programmed rather than make changes to the Linux kernel in order to make scalability improvements. This may indicate that what has come before is done quite well when considering that the Linux kernel optimizations showed more improvement when put in conjunction with the application improvements.

That is just the software that we can change, The programs are also limited by IO hardware.

==Critique==
Aside from a few acronyms that were unexplained (e.g. Linux TLB) the paper has no real stylistic problems.

===memcached: ''Section 5.3''===
memcached is treated with near perfect fairness in the paper. Its an in-memory service, so the ignored storage IO bottleneck does not affect it at all. Likewise the "stock" and "PK" implementations are given the same test suite, so there is no advantage given to either. memcached itself is non-scalable, so the MIT team was forced to run one instance per-core to keep up throughput. The FAQ at memcached.org's wiki suggests using multiple implementations per-server as a work around to another problem, which implies that there is no great problem with running multiple servers on one machine [3].

===Apache: ''Section 5.4''===
Linux has a built in kernel flaw where network packets are forced to travel though multiple queues before they arrive at queue where they can be processed by the application. This imposes significant costs on multi-core systems due to queue locking costs. This flaw inherently diminishes the performance of Apache on multi-core system due to multiple threads spread across cores being forced to deal with these mutex (mutual exclusion) costs. For the sake of this experiment Apache had a separate instance on every core listening on different ports which is not a practical real world application but merely an attempt to implement better parallel execution on a traditional kernel. The patched kernel implementation of the network stack is also specific to the problem at hand, which is processing multiple short lived connections across multiple cores. Although this provides a performance increase in the given scenario, in more general applications network performance might suffer. These tests were also rigged to avoid bottlenecks in place by network and file storage hardware. Meaning, making the proposed modifications to the kernel wont necessarily produce the same increase in productivity as described in the article. This is very much evident in the test where performance degrades past 36 cores due to limitation of the networking hardware.

*not the same thing. The article you pointed out talks about spawning different processes to accept connections which apache already does by default. what the article says is they had a separate instance of apache on each core each capable of spawning those processes. if you read over section 3.3 again u'll notice it says apache spawns a process for each incoming connection

===gmake: ''Section 5.6''===
Since the inherent nature of gmake makes it quite parallel, the testing and updating that were attempted on gmake resulted in essentially the same scalability results for both the stock and modified kernel. The only change that was found was that gmake spent slightly less time at the system level because of the changes that were made to the system's caching. As stated in the paper, the execution time of gmake relies quite heavily on the compiler that is uses with gmake, so depending on which compiler was chosen, gmake could run worse or even slightly better. In any case, there seems to be no fairness concerns when it comes to the scalability testing of gmake as the same application load-out was used for all of the tests.

===Conclusion: ''Sections 6 & 7''===
Given that all tests are more or less fair for the purposes of the benchmarks. They would support the Hypothesis that Linux can be made to scale, at least to 48 cores. Thus the conclusion is fair iff the rest of the paper is fair.

Now you just have to fill in how fair the rest of the paper is.

==References==

[3] memcached's wiki: http://code.google.com/p/memcached/wiki/FAQ#Can_I_use_different_size_caches_across_servers_and_will_memcache

You will almost certainly have to refer to other resources; please cite these resources in the style of citation of the papers assigned (inlined numbered references). Place your bibliographic entries in this section.

'''the paper itself doesn't need to be referenced more than once as this is a critique of the paper...'''
[1] Silas Boyd-Wickizer et al. "An Analysis of Linux Scalability to Many Cores". In ''OSDI '10, 9th USENIX Symposium on OS Design and Implementation'', Vancouver, BC, Canada, 2010. http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf.

gmake:

[http://www.gnu.org/software/make/manual/make.html gmake Manual]

[http://www.gnu.org/software/make/ gmake Main Page]

==Deprecated==
===Background Concepts===
* Exim: ''Section 3.1'':
**Exim is a mail server for Unix. It's fairly parallel. The server forks a new process for each connection and twice to deliver each message. It spends 69% of its time in the kernel on a single core.

* PostgreSQL: ''Section 3.4'':
**As implied by the name PostgreSQL is a SQL database. PostgreSQL starts a separate process for each connection and uses kernel locking interfaces extensively to provide concurrent access to the database. Due to bottlenecks introduced in its code and in the kernel code, the amount of time PostgreSQL spends in the kernel increases very rapidly with addition of new cores. On a single core system PostgreSQL spends only 1.5% of its time in the kernel. On a 48 core system the execution time in the kernel jumps to 82%.

Talk:COMP 3000 Essay 2 2010 Question 1

2010-12-03T02:42:35Z

Kkashigi: /* Apache: Section 5.4 */

=Class and Notices=
(Nov. 30, 2010) Prof. Anil stated that we should focus on the 3 easiest to understand parts in section 5 and elaborate on them.

- Also, I, Daniel B., work Thursday night, so I will be finishing up as much of my part as I can for the essay before Thursday morning's class, then maybe we can all meet up in a lab in HP and put the finishing touches on the essay. I will be available online Wednesday night from about 6:30pm onwards and will be in the game dev lab or CCSS lounge Wednesday morning from about 11am to 2pm if anyone would like to meet up with me at those times.

- [[I suggest we meet up Thursday morning after Operating Systems in order to discuss and finalize the essay. Maybe we can even designate a lab for the group to meet up in. Any suggestions?]] - Daniel B.

- HP 3115 since there wont be a class in there (as its our tutorial and we know there won't be anyone there)
-- Go to Wireless Lab next to CCSS Lounge. Andrew and Dan B. will be there.

- If its all the same to you guys mind if I just join you via msn or iirc? Or phone if you really want. -Rannath

- I'm working today, but I'll be at a computer reading this page/contributing to my section. Depending on how busy I am, I should be able to get some significant writing in before 4pm today on my section and any additional sections required. RP

- I wont be there either. that does not mean i wont/cant contribute. I'll be on msn or you can just email me. -kirill

=Group members=
Patrick Young [mailto:Rannath@gmail.com Rannath@gmail.com]

Daniel Beimers [mailto:demongyro@gmail.com demongyro@gmail.com]

Andrew Bown [mailto:abown2@connect.carleton.ca abown2@connect.carleton.ca]

Kirill Kashigin [mailto:k.kashigin@gmail.com k.kashigin@gmail.com]

Rovic Perdon [mailto:rperdon@gmail.com rperdon@gmail.com]

Daniel Sont [mailto:dan.sont@gmail.com dan.sont@gmail.com]

=Methodology=
We should probably have our work verified by at least one group member before posting to the actual page

=To Do=
*contribution (conclusion mostly): just need to re-work some sections and follow the cues I left in the Conclusion section.
*critique (conclusion mostly): critique the conclusion of the essay
*style: the style section is largely untouched. Daniel and I([[Rannath]]) have puts some thoughts there, but that section needs to be made into sentences.

=Essay=
==Paper - DONE!!!==
This paper was authored by - Silas Boyd-Wickizer, Austin T. Clements, Yandong Mao, Aleksey Pesterev, M. Frans Kaashoek, Robert Morris, and Nickolai Zeldovich.

They all work at MIT CSAIL.

The paper: [http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf An Analysis of Linux Scalability to Many Cores]

==Background Concepts - DONE!!!==
===memcached: ''Section 3.2''===
memcached is an in-memory hash table server. One instance of memcached running on many different cores is bottlenecked by an internal lock, which is avoided by the MIT team by running one instance per core. Clients each connect to a single instance of memcached, allowing the server to simulate parallelism without needing to make major changes to the application or kernel. With few requests, memcached spends 80% of its time in the kernel on one core, mostly processing packets.[1]

===Apache: ''Section 3.3''===
Apache is a web server that has been used in previous Linux scalability studies. In the case of this study, Apache has been configured to run a separate process on each core. Each process, in turn, has multiple threads (making it a perfect example of parallel programming). Each process uses one of their threads to accepting incoming connections and others are used to process these connections. On a single core processor, Apache spends 60% of its execution time in the kernel.[1]

===gmake: ''Section 3.5''===
gmake is an unofficial default benchmark in the Linux community which is used in this paper to build the Linux kernel. gmake takes a file called a makefile and processes its recipes for the requisite files to determine how and when to remake or recompile code. With a simple command -j or --jobs, gmake can process many of these recipes in parallel. Since gmake creates more processes than cores, it can make proper use of multiple cores to process the recipes.[2] Since gmake involves much reading and writing, in order to prevent bottlenecks due to the filesystem or hardware, the test cases use an in-memory filesystem tmpfs, which gives them a backdoor around the bottlenecks for testing purposes. In addition to this, gmake is limited in scalability due to the serial processes that run at the beginning and end of its execution, which limits its scalability to a small degree. gmake spends much of its execution time with its compiler, processing the recipes and recompiling code, but still spend 7.6% of its time in system time.[1]

[2] http://www.gnu.org/software/make/manual/make.html

==Research problem - DONE!!!==
As technology progresses, the number of core a main processor can have is increasing at an impressive rate. Soon personal computers will have so many cores that scalability will be an issue. There has to be a way that standard user level Linux kernel will scale with a 48-core system[[#Foot1|1]]. The problem with a standard Linux OS is that they are not designed for massive scalability, which will soon prove to be a problem. The issue with scalability is that a solo core will perform much more work compared to a single core working with 47 other cores. Although traditional logic states that the situation makes sense because there are 48 cores dividing the work, the information should be processed as fast as possible with each core doing as much work as possible.

To fix those scalability issues, it is necessary to focus on three major areas: the Linux kernel, user level design and how applications use kernel services. The Linux kernel can be improved by optimizing sharing and use the current advantages of recent improvement to scalability features. On the user level design, applications can be improved so that there is more focus on parallelism since some programs have not implemented those improved features. The final aspect of improving scalability is how an application uses kernel services to better share resources so that different aspects of the program are not conflicting over the same services. All of the bottlenecks are found easily and actually only take simple changes to correct or avoid.[[#Foot1|1]]

This research uses a foundation of previous research discovered during the development of scalability in UNIX systems. The major developments from shared memory machines[[#Foot2|2]] and wait-free synchronization to fast message passing ended up creating a base set of techniques, which can be used to improve scalability. These techniques have been incorporated in all major operating system including Linux, Mac OS X and Windows. Linux has been improved with kernel subsystems, such as Read-Copy-Update, which is an algorithm that is used to avoid locks and atomic instructions which affect scalability.[[#Foot3|3]] There is an excellent base of research on Linux scalability studies that have already been written, on which this research paper can model its testing standards. These papers include research on improving scalability on a 32-core machine.[[#Foot4|4]] In addition, the base of studies can be used to improve the results of these experiments by learning from the previous results. This research may also aid in identifying bottlenecks which speed up creating solutions for those problems.

==Contribution==
All contributions in this paper are the result of the identification and removal or marginalization of bottlenecks.

===What hinders scalability: ''Section 4.1''===
*The percentage of serialization in a program has a lot to do with how much an application can be sped up. This is Amdahl's Law
** Amdahl's Law states that a parallel program can only be sped up by the inverse of the proportion of the program that cannot be made parallel (e.g. 25%(.25) non-parallel --> limit of 4x speedup) (I can't get this to sound right someone fix it please -[[Rannath]] <- I will fix [[Daniel B.]]
*Types of serializing interactions found in the MOSBENCH apps:
**Locking of shared data structure as the number of cores increase leads to an increase in lock wait time
**Writing to shared memory as the number of cores increase leads to an increase in the execution time of the cache coherence protocol
**Competing for space in shared hardware cache as the number of cores increase leads to an increase in cache miss rate
**Competing for shared hardware resources as the number of cores increase leads to time lost waiting for resources
**Not enough tasks for cores leads to idle cores

===Multicore packet processing: ''Section 4.2''===
Linux packet processing technique requires the packets to travel along several queues before it finally becomes available for the application to use. This technique works well for most general socket applications. In recent kernel releases Linux takes advantage of multiple hardware queues (when available on the given network interface) or Receive Packet Steering[1] to direct packet flow onto different cores for processing. Or even go as far as directing packet flow to the core on which the application is running using Receive Flow Steering[2] for even better performance. Linux also attempts to increase performance using a sampling technique where it checks every 20th outgoing packet and directs flow based on its hash. This poses a problem for short lived connections like those associated with Apache since there is great potential for packets to be misdirected.

In general this technique performs poorly when there are numerous open connections spread across multiple cores due to mutex (mutual exclusion) delays and cache misses. In such scenarios its better to process all connections, with associated packets and queues, on one core to avoid said issues. The patched kernel's implementation proposed in this article uses multiple hardware queues (which can be accomplished through Receive Packet Sharing) to direct all packets from a given connection to the same core. In turn Apache is modified to only accept connections if the thread dedicated to processing them is on the same core. If the current core's queue is found to be empty it will attempt to obtain work from queues located on different cores. This configuration is ideal for numerous short connections as all the work for them in accomplished quickly on one core avoiding unnecessary mutex delays associated with packet queues and inter-core cache misses.

[1] J. Corbet. Receive Packet Steering, November 2009. http://lwn.net/Articles/362339/.

[2] J. Edge. Receive Flow Steering, April 2010. http://lwn.net/Articles/382428/.

===Sloppy counters: ''Section 4.3''===
Bottlenecks were encountered when the applications undergoing testing were referencing and updating shared counters for multiple cores. The solution in the paper is to use sloppy counters, which gets each core to track its own separate counts of references and uses a central shared counter to keep all counts on track. This is ideal because each core updates its counts by modifying its per-core counter, usually only needing access to its own local cache, cutting down on waiting for locks or serialization. Sloppy counters are also backwards-compatible with existing shared-counter code, making its implementation much easier to accomplish. The main disadvantages of the sloppy counters are that in situations where object de-allocation occurs often, because the de-allocation itself is an expensive operation, and the counters use up space proportional to the number of cores.

===Lock-free comparison & Avoiding unnecessary locking: ''Section 4.4 & 4.7''===
The traditional Linux kernel has very low scalability for name lookups in the directory entry cache. This means there is reduced performance in returning information pertaining to a specific file path when there are multiple threads trying to access files in common parent directories due to the kernel serializing the process. This problem is solved in the patched kernel by introducing a new counter to keep track of threads actively looking at the directory entry cache. If a certain thread threatens an entry currently in use by another, the default locking protocol is used to avoid race conditions. If the activities have no bearing on each other the situation is rightfully ignored allowing for much faster access to different different entries in the directory entry cache.

There are many other locks/mutexes that have special cases where they don't need to lock. Others can be split from locking the whole data structure to locking a part of it. Both these changes remove or reduce bottlenecks.

===Per-Core Data Structures: ''Section 4.5''===
Three centralized data structures were causing bottlenecks due to lock contention - a per-superblock list of open files, vfsmount table, the packet buffers free list. Each data structure was decentralized to per-core versions of itself. In the case of vfsmount the central data structure was maintained, and any per-core misses got written from the central table to the per-core table.

===Eliminating false sharing: ''Section 4.6''===
Misplaced variables on the cache cause different cores to request the same line, but different variables. With two different cores requesting the same variable one reading, the other writing there was a severe bottleneck. By moving the often written variable to another line the bottleneck was removed.

===memcached: ''Section 5.3''===
memcached nearly doubles throughput at 48 cores on the paper's implementation over the stock implementation. After improvements to the Linux kernel memcached is limited by hardware. Improvements to hardware scalability, virtual queue handling in this case, will allow further improvements to memcached.

===Apache: ''Section 5.4''===
Apache keeps fairly a equal amount of throughput up to 36 cores. Then it slopes downwards. At 48 cores it still has an improved throughput of more than 12 times. Apache like memcached is limited by hardware. At higher core count the network card simply cannot handle the number of packets, and the FIFO queue it holds for them overflows.

===gmake: ''Section 5.6''===
gmakes run time is nearly unchanged by the implementation changes presented in this paper. This is largely because the program has serial sections of code and some processes that finish somewhat later then all others, which prevents perfect scalability. Even then it achieves the greatest level of scalability out of the three programs (35X speed up for 48 cores.)

===Conclusion: ''Sections 6 & 7''===
The contribution of this paper is a lot of research that has focus upon techniques and methods for scalability. This is accomplished through programming of applications alongside kernel programming. This research contributes by evaluating the scalability discrepancies of applications programming and kernel programming. Key discoveries in this research show the effectiveness of the kernel in handling scaling amongst CPU cores. In looking at the issue of scalability it is important to note the causes of the factors which hinder scalability.

It has been shown that simple scaling techniques can be effective in increasing scalability. The authors looked at three different approaches to removing the bottlenecks within the system. The first was to see if there were issues within the linux kernel application, the second was to identify issues with the application design and the third was to address how the application interacts with the linux kernel services. Through this approach, the authors were able to quickly identify problems such as bottlenecks and apply simple techniques in fixing the issues at hand to reap some beneficial aspects. Some of the sections listed provide insight into the improvements that can be reaped from these optimizations.

Through this research on various techniques as listed above, it was determined by the authors that the Linux kernel itself has many incorporated techniques used to improve scalability. The authors actually go on to speculate that "perhaps it is the case that Linux's system-call API is well suited to an implementation that avoids unnecessary contention over kernel objects." This tends to show that the work in the Linux community has improved Linux a large amount and is current with modern techniques for optimization. It could also be interpreted from the paper that it may be to the benefit of the community to change how the applications are programmed rather than make changes to the Linux kernel in order to make scalability improvements. This may indicate that what has come before is done quite well when considering that the Linux kernel optimizations showed more improvement when put in conjunction with the application improvements.

That is just the software that we can change, The programs are also limited by IO hardware.

==Critique==
Aside from a few acronyms that were unexplained (e.g. Linux TLB) the paper has no real stylistic problems.

===memcached: ''Section 5.3''===
memcached is treated with near perfect fairness in the paper. Its an in-memory service, so the ignored storage IO bottleneck does not affect it at all. Likewise the "stock" and "PK" implementations are given the same test suite, so there is no advantage given to either. memcached itself is non-scalable, so the MIT team was forced to run one instance per-core to keep up throughput. The FAQ at memcached.org's wiki suggests using multiple implementations per-server as a work around to another problem, which implies that there is no great problem with running multiple servers on one machine [3].

===Apache: ''Section 5.4''===
Linux has a built in kernel flaw where network packets are forced to travel though multiple queues before they arrive at queue where they can be processed by the application. This imposes significant costs on multi-core systems due to queue locking costs. This flaw inherently diminishes the performance of Apache on multi-core system due to multiple threads spread across cores being forced to deal with these mutex (mutual exclusion) costs. The patched kernel implementation of the network stack is also specific to the problem at hand, which is processing multiple short lived connections across multiple cores. Although this provides a performance increase in the given scenario, in more general applications network performance might suffer. These tests were also rigged to avoid bottlenecks in place by network and file storage hardware. Meaning, making the proposed modifications to the kernel wont necessarily produce the same increase in productivity as described in the article. This is very much evident in the test where performance degrades past 36 cores due to limitation of the networking hardware.

''For the sake of this experiment Apache had a separate instance on every core listening on different ports which is not a practical real world application but merely an attempt to implement better parallel execution on a traditional kernel. ''
*This is untrue, the '''instance''' spawned one '''process''' per-core
*"Thus, for stock Linux, we run a separate instance of Apache per core with each server running on a distinct port" second sentence into 5.4 -kirill

===gmake: ''Section 5.6''===
Since the inherent nature of gmake makes it quite parallel, the testing and updating that were attempted on gmake resulted in essentially the same scalability results for both the stock and modified kernel. The only change that was found was that gmake spent slightly less time at the system level because of the changes that were made to the system's caching. As stated in the paper, the execution time of gmake relies quite heavily on the compiler that is uses with gmake, so depending on which compiler was chosen, gmake could run worse or even slightly better. In any case, there seems to be no fairness concerns when it comes to the scalability testing of gmake as the same application load-out was used for all of the tests.

===Conclusion: ''Sections 6 & 7''===
Given that all tests are more or less fair for the purposes of the benchmarks. They would support the Hypothesis that Linux can be made to scale, at least to 48 cores. Thus the conclusion is fair iff the rest of the paper is fair.

Now you just have to fill in how fair the rest of the paper is.

==References==

[3] memcached's wiki: http://code.google.com/p/memcached/wiki/FAQ#Can_I_use_different_size_caches_across_servers_and_will_memcache

You will almost certainly have to refer to other resources; please cite these resources in the style of citation of the papers assigned (inlined numbered references). Place your bibliographic entries in this section.

'''the paper itself doesn't need to be referenced more than once as this is a critique of the paper...'''
[1] Silas Boyd-Wickizer et al. "An Analysis of Linux Scalability to Many Cores". In ''OSDI '10, 9th USENIX Symposium on OS Design and Implementation'', Vancouver, BC, Canada, 2010. http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf.

gmake:

[http://www.gnu.org/software/make/manual/make.html gmake Manual]

[http://www.gnu.org/software/make/ gmake Main Page]

==Deprecated==
===Background Concepts===
* Exim: ''Section 3.1'':
**Exim is a mail server for Unix. It's fairly parallel. The server forks a new process for each connection and twice to deliver each message. It spends 69% of its time in the kernel on a single core.

* PostgreSQL: ''Section 3.4'':
**As implied by the name PostgreSQL is a SQL database. PostgreSQL starts a separate process for each connection and uses kernel locking interfaces extensively to provide concurrent access to the database. Due to bottlenecks introduced in its code and in the kernel code, the amount of time PostgreSQL spends in the kernel increases very rapidly with addition of new cores. On a single core system PostgreSQL spends only 1.5% of its time in the kernel. On a 48 core system the execution time in the kernel jumps to 82%.

Talk:COMP 3000 Essay 2 2010 Question 1

2010-12-03T02:40:15Z

Kkashigi: /* Apache: Section 5.4 */

=Class and Notices=
(Nov. 30, 2010) Prof. Anil stated that we should focus on the 3 easiest to understand parts in section 5 and elaborate on them.

- Also, I, Daniel B., work Thursday night, so I will be finishing up as much of my part as I can for the essay before Thursday morning's class, then maybe we can all meet up in a lab in HP and put the finishing touches on the essay. I will be available online Wednesday night from about 6:30pm onwards and will be in the game dev lab or CCSS lounge Wednesday morning from about 11am to 2pm if anyone would like to meet up with me at those times.

- [[I suggest we meet up Thursday morning after Operating Systems in order to discuss and finalize the essay. Maybe we can even designate a lab for the group to meet up in. Any suggestions?]] - Daniel B.

- HP 3115 since there wont be a class in there (as its our tutorial and we know there won't be anyone there)
-- Go to Wireless Lab next to CCSS Lounge. Andrew and Dan B. will be there.

- If its all the same to you guys mind if I just join you via msn or iirc? Or phone if you really want. -Rannath

- I'm working today, but I'll be at a computer reading this page/contributing to my section. Depending on how busy I am, I should be able to get some significant writing in before 4pm today on my section and any additional sections required. RP

- I wont be there either. that does not mean i wont/cant contribute. I'll be on msn or you can just email me. -kirill

=Group members=
Patrick Young [mailto:Rannath@gmail.com Rannath@gmail.com]

Daniel Beimers [mailto:demongyro@gmail.com demongyro@gmail.com]

Andrew Bown [mailto:abown2@connect.carleton.ca abown2@connect.carleton.ca]

Kirill Kashigin [mailto:k.kashigin@gmail.com k.kashigin@gmail.com]

Rovic Perdon [mailto:rperdon@gmail.com rperdon@gmail.com]

Daniel Sont [mailto:dan.sont@gmail.com dan.sont@gmail.com]

=Methodology=
We should probably have our work verified by at least one group member before posting to the actual page

=To Do=
*contribution (conclusion mostly): just need to re-work some sections and follow the cues I left in the Conclusion section.
*critique (conclusion mostly): critique the conclusion of the essay
*style: the style section is largely untouched. Daniel and I([[Rannath]]) have puts some thoughts there, but that section needs to be made into sentences.

=Essay=
==Paper - DONE!!!==
This paper was authored by - Silas Boyd-Wickizer, Austin T. Clements, Yandong Mao, Aleksey Pesterev, M. Frans Kaashoek, Robert Morris, and Nickolai Zeldovich.

They all work at MIT CSAIL.

The paper: [http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf An Analysis of Linux Scalability to Many Cores]

==Background Concepts - DONE!!!==
===memcached: ''Section 3.2''===
memcached is an in-memory hash table server. One instance of memcached running on many different cores is bottlenecked by an internal lock, which is avoided by the MIT team by running one instance per core. Clients each connect to a single instance of memcached, allowing the server to simulate parallelism without needing to make major changes to the application or kernel. With few requests, memcached spends 80% of its time in the kernel on one core, mostly processing packets.[1]

===Apache: ''Section 3.3''===
Apache is a web server that has been used in previous Linux scalability studies. In the case of this study, Apache has been configured to run a separate process on each core. Each process, in turn, has multiple threads (making it a perfect example of parallel programming). Each process uses one of their threads to accepting incoming connections and others are used to process these connections. On a single core processor, Apache spends 60% of its execution time in the kernel.[1]

===gmake: ''Section 3.5''===
gmake is an unofficial default benchmark in the Linux community which is used in this paper to build the Linux kernel. gmake takes a file called a makefile and processes its recipes for the requisite files to determine how and when to remake or recompile code. With a simple command -j or --jobs, gmake can process many of these recipes in parallel. Since gmake creates more processes than cores, it can make proper use of multiple cores to process the recipes.[2] Since gmake involves much reading and writing, in order to prevent bottlenecks due to the filesystem or hardware, the test cases use an in-memory filesystem tmpfs, which gives them a backdoor around the bottlenecks for testing purposes. In addition to this, gmake is limited in scalability due to the serial processes that run at the beginning and end of its execution, which limits its scalability to a small degree. gmake spends much of its execution time with its compiler, processing the recipes and recompiling code, but still spend 7.6% of its time in system time.[1]

[2] http://www.gnu.org/software/make/manual/make.html

==Research problem - DONE!!!==
As technology progresses, the number of core a main processor can have is increasing at an impressive rate. Soon personal computers will have so many cores that scalability will be an issue. There has to be a way that standard user level Linux kernel will scale with a 48-core system[[#Foot1|1]]. The problem with a standard Linux OS is that they are not designed for massive scalability, which will soon prove to be a problem. The issue with scalability is that a solo core will perform much more work compared to a single core working with 47 other cores. Although traditional logic states that the situation makes sense because there are 48 cores dividing the work, the information should be processed as fast as possible with each core doing as much work as possible.

To fix those scalability issues, it is necessary to focus on three major areas: the Linux kernel, user level design and how applications use kernel services. The Linux kernel can be improved by optimizing sharing and use the current advantages of recent improvement to scalability features. On the user level design, applications can be improved so that there is more focus on parallelism since some programs have not implemented those improved features. The final aspect of improving scalability is how an application uses kernel services to better share resources so that different aspects of the program are not conflicting over the same services. All of the bottlenecks are found easily and actually only take simple changes to correct or avoid.[[#Foot1|1]]

This research uses a foundation of previous research discovered during the development of scalability in UNIX systems. The major developments from shared memory machines[[#Foot2|2]] and wait-free synchronization to fast message passing ended up creating a base set of techniques, which can be used to improve scalability. These techniques have been incorporated in all major operating system including Linux, Mac OS X and Windows. Linux has been improved with kernel subsystems, such as Read-Copy-Update, which is an algorithm that is used to avoid locks and atomic instructions which affect scalability.[[#Foot3|3]] There is an excellent base of research on Linux scalability studies that have already been written, on which this research paper can model its testing standards. These papers include research on improving scalability on a 32-core machine.[[#Foot4|4]] In addition, the base of studies can be used to improve the results of these experiments by learning from the previous results. This research may also aid in identifying bottlenecks which speed up creating solutions for those problems.

==Contribution==
All contributions in this paper are the result of the identification and removal or marginalization of bottlenecks.

===What hinders scalability: ''Section 4.1''===
*The percentage of serialization in a program has a lot to do with how much an application can be sped up. This is Amdahl's Law
** Amdahl's Law states that a parallel program can only be sped up by the inverse of the proportion of the program that cannot be made parallel (e.g. 25%(.25) non-parallel --> limit of 4x speedup) (I can't get this to sound right someone fix it please -[[Rannath]] <- I will fix [[Daniel B.]]
*Types of serializing interactions found in the MOSBENCH apps:
**Locking of shared data structure as the number of cores increase leads to an increase in lock wait time
**Writing to shared memory as the number of cores increase leads to an increase in the execution time of the cache coherence protocol
**Competing for space in shared hardware cache as the number of cores increase leads to an increase in cache miss rate
**Competing for shared hardware resources as the number of cores increase leads to time lost waiting for resources
**Not enough tasks for cores leads to idle cores

===Multicore packet processing: ''Section 4.2''===
Linux packet processing technique requires the packets to travel along several queues before it finally becomes available for the application to use. This technique works well for most general socket applications. In recent kernel releases Linux takes advantage of multiple hardware queues (when available on the given network interface) or Receive Packet Steering[1] to direct packet flow onto different cores for processing. Or even go as far as directing packet flow to the core on which the application is running using Receive Flow Steering[2] for even better performance. Linux also attempts to increase performance using a sampling technique where it checks every 20th outgoing packet and directs flow based on its hash. This poses a problem for short lived connections like those associated with Apache since there is great potential for packets to be misdirected.

In general this technique performs poorly when there are numerous open connections spread across multiple cores due to mutex (mutual exclusion) delays and cache misses. In such scenarios its better to process all connections, with associated packets and queues, on one core to avoid said issues. The patched kernel's implementation proposed in this article uses multiple hardware queues (which can be accomplished through Receive Packet Sharing) to direct all packets from a given connection to the same core. In turn Apache is modified to only accept connections if the thread dedicated to processing them is on the same core. If the current core's queue is found to be empty it will attempt to obtain work from queues located on different cores. This configuration is ideal for numerous short connections as all the work for them in accomplished quickly on one core avoiding unnecessary mutex delays associated with packet queues and inter-core cache misses.

[1] J. Corbet. Receive Packet Steering, November 2009. http://lwn.net/Articles/362339/.

[2] J. Edge. Receive Flow Steering, April 2010. http://lwn.net/Articles/382428/.

===Sloppy counters: ''Section 4.3''===
Bottlenecks were encountered when the applications undergoing testing were referencing and updating shared counters for multiple cores. The solution in the paper is to use sloppy counters, which gets each core to track its own separate counts of references and uses a central shared counter to keep all counts on track. This is ideal because each core updates its counts by modifying its per-core counter, usually only needing access to its own local cache, cutting down on waiting for locks or serialization. Sloppy counters are also backwards-compatible with existing shared-counter code, making its implementation much easier to accomplish. The main disadvantages of the sloppy counters are that in situations where object de-allocation occurs often, because the de-allocation itself is an expensive operation, and the counters use up space proportional to the number of cores.

===Lock-free comparison & Avoiding unnecessary locking: ''Section 4.4 & 4.7''===
The traditional Linux kernel has very low scalability for name lookups in the directory entry cache. This means there is reduced performance in returning information pertaining to a specific file path when there are multiple threads trying to access files in common parent directories due to the kernel serializing the process. This problem is solved in the patched kernel by introducing a new counter to keep track of threads actively looking at the directory entry cache. If a certain thread threatens an entry currently in use by another, the default locking protocol is used to avoid race conditions. If the activities have no bearing on each other the situation is rightfully ignored allowing for much faster access to different different entries in the directory entry cache.

There are many other locks/mutexes that have special cases where they don't need to lock. Others can be split from locking the whole data structure to locking a part of it. Both these changes remove or reduce bottlenecks.

===Per-Core Data Structures: ''Section 4.5''===
Three centralized data structures were causing bottlenecks due to lock contention - a per-superblock list of open files, vfsmount table, the packet buffers free list. Each data structure was decentralized to per-core versions of itself. In the case of vfsmount the central data structure was maintained, and any per-core misses got written from the central table to the per-core table.

===Eliminating false sharing: ''Section 4.6''===
Misplaced variables on the cache cause different cores to request the same line, but different variables. With two different cores requesting the same variable one reading, the other writing there was a severe bottleneck. By moving the often written variable to another line the bottleneck was removed.

===memcached: ''Section 5.3''===
memcached nearly doubles throughput at 48 cores on the paper's implementation over the stock implementation. After improvements to the Linux kernel memcached is limited by hardware. Improvements to hardware scalability, virtual queue handling in this case, will allow further improvements to memcached.

===Apache: ''Section 5.4''===
Apache keeps fairly a equal amount of throughput up to 36 cores. Then it slopes downwards. At 48 cores it still has an improved throughput of more than 12 times. Apache like memcached is limited by hardware. At higher core count the network card simply cannot handle the number of packets, and the FIFO queue it holds for them overflows.

===gmake: ''Section 5.6''===
gmakes run time is nearly unchanged by the implementation changes presented in this paper. This is largely because the program has serial sections of code and some processes that finish somewhat later then all others, which prevents perfect scalability. Even then it achieves the greatest level of scalability out of the three programs (35X speed up for 48 cores.)

===Conclusion: ''Sections 6 & 7''===
The contribution of this paper is a lot of research that has focus upon techniques and methods for scalability. This is accomplished through programming of applications alongside kernel programming. This research contributes by evaluating the scalability discrepancies of applications programming and kernel programming. Key discoveries in this research show the effectiveness of the kernel in handling scaling amongst CPU cores. In looking at the issue of scalability it is important to note the causes of the factors which hinder scalability.

It has been shown that simple scaling techniques can be effective in increasing scalability. The authors looked at three different approaches to removing the bottlenecks within the system. The first was to see if there were issues within the linux kernel application, the second was to identify issues with the application design and the third was to address how the application interacts with the linux kernel services. Through this approach, the authors were able to quickly identify problems such as bottlenecks and apply simple techniques in fixing the issues at hand to reap some beneficial aspects. Some of the sections listed provide insight into the improvements that can be reaped from these optimizations.

Through this research on various techniques as listed above, it was determined by the authors that the Linux kernel itself has many incorporated techniques used to improve scalability. The authors actually go on to speculate that "perhaps it is the case that Linux's system-call API is well suited to an implementation that avoids unnecessary contention over kernel objects." This tends to show that the work in the Linux community has improved Linux a large amount and is current with modern techniques for optimization. It could also be interpreted from the paper that it may be to the benefit of the community to change how the applications are programmed rather than make changes to the Linux kernel in order to make scalability improvements. This may indicate that what has come before is done quite well when considering that the Linux kernel optimizations showed more improvement when put in conjunction with the application improvements.

That is just the software that we can change, The programs are also limited by IO hardware.

==Critique==
Aside from a few acronyms that were unexplained (e.g. Linux TLB) the paper has no real stylistic problems.

===memcached: ''Section 5.3''===
memcached is treated with near perfect fairness in the paper. Its an in-memory service, so the ignored storage IO bottleneck does not affect it at all. Likewise the "stock" and "PK" implementations are given the same test suite, so there is no advantage given to either. memcached itself is non-scalable, so the MIT team was forced to run one instance per-core to keep up throughput. The FAQ at memcached.org's wiki suggests using multiple implementations per-server as a work around to another problem, which implies that there is no great problem with running multiple servers on one machine [3].

===Apache: ''Section 5.4''===
Linux has a built in kernel flaw where network packets are forced to travel though multiple queues before they arrive at queue where they can be processed by the application. This imposes significant costs on multi-core systems due to queue locking costs. This flaw inherently diminishes the performance of Apache on multi-core system due to multiple threads spread across cores being forced to deal with these mutex (mutual exclusion) costs. The patched kernel implementation of the network stack is also specific to the problem at hand, which is processing multiple short lived connections across multiple cores. Although this provides a performance increase in the given scenario, in more general applications network performance might suffer. These tests were also rigged to avoid bottlenecks in place by network and file storage hardware. Meaning, making the proposed modifications to the kernel wont necessarily produce the same increase in productivity as described in the article. This is very much evident in the test where performance degrades past 36 cores due to limitation of the networking hardware.

''For the sake of this experiment Apache had a separate instance on every core listening on different ports which is not a practical real world application but merely an attempt to implement better parallel execution on a traditional kernel. ''
*This is untrue, the '''instance''' spawned one '''process''' per-core
*"Thus, for stock Linux, we run a separate of Apache per core with each server running on a distinct port" second sentance into 5.4 -kirill

===gmake: ''Section 5.6''===
Since the inherent nature of gmake makes it quite parallel, the testing and updating that were attempted on gmake resulted in essentially the same scalability results for both the stock and modified kernel. The only change that was found was that gmake spent slightly less time at the system level because of the changes that were made to the system's caching. As stated in the paper, the execution time of gmake relies quite heavily on the compiler that is uses with gmake, so depending on which compiler was chosen, gmake could run worse or even slightly better. In any case, there seems to be no fairness concerns when it comes to the scalability testing of gmake as the same application load-out was used for all of the tests.

===Conclusion: ''Sections 6 & 7''===
Given that all tests are more or less fair for the purposes of the benchmarks. They would support the Hypothesis that Linux can be made to scale, at least to 48 cores. Thus the conclusion is fair iff the rest of the paper is fair.

Now you just have to fill in how fair the rest of the paper is.

==References==

[3] memcached's wiki: http://code.google.com/p/memcached/wiki/FAQ#Can_I_use_different_size_caches_across_servers_and_will_memcache

You will almost certainly have to refer to other resources; please cite these resources in the style of citation of the papers assigned (inlined numbered references). Place your bibliographic entries in this section.

'''the paper itself doesn't need to be referenced more than once as this is a critique of the paper...'''
[1] Silas Boyd-Wickizer et al. "An Analysis of Linux Scalability to Many Cores". In ''OSDI '10, 9th USENIX Symposium on OS Design and Implementation'', Vancouver, BC, Canada, 2010. http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf.

gmake:

[http://www.gnu.org/software/make/manual/make.html gmake Manual]

[http://www.gnu.org/software/make/ gmake Main Page]

==Deprecated==
===Background Concepts===
* Exim: ''Section 3.1'':
**Exim is a mail server for Unix. It's fairly parallel. The server forks a new process for each connection and twice to deliver each message. It spends 69% of its time in the kernel on a single core.

* PostgreSQL: ''Section 3.4'':
**As implied by the name PostgreSQL is a SQL database. PostgreSQL starts a separate process for each connection and uses kernel locking interfaces extensively to provide concurrent access to the database. Due to bottlenecks introduced in its code and in the kernel code, the amount of time PostgreSQL spends in the kernel increases very rapidly with addition of new cores. On a single core system PostgreSQL spends only 1.5% of its time in the kernel. On a 48 core system the execution time in the kernel jumps to 82%.

Talk:COMP 3000 Essay 2 2010 Question 1

2010-12-03T02:01:39Z

Kkashigi: /* Lock-free comparison: Section 4.4 */

=Class and Notices=
(Nov. 30, 2010) Prof. Anil stated that we should focus on the 3 easiest to understand parts in section 5 and elaborate on them.

- Also, I, Daniel B., work Thursday night, so I will be finishing up as much of my part as I can for the essay before Thursday morning's class, then maybe we can all meet up in a lab in HP and put the finishing touches on the essay. I will be available online Wednesday night from about 6:30pm onwards and will be in the game dev lab or CCSS lounge Wednesday morning from about 11am to 2pm if anyone would like to meet up with me at those times.

- [[I suggest we meet up Thursday morning after Operating Systems in order to discuss and finalize the essay. Maybe we can even designate a lab for the group to meet up in. Any suggestions?]] - Daniel B.

- HP 3115 since there wont be a class in there (as its our tutorial and we know there won't be anyone there)
-- Go to Wireless Lab next to CCSS Lounge. Andrew and Dan B. will be there.

- If its all the same to you guys mind if I just join you via msn or iirc? Or phone if you really want. -Rannath

- I'm working today, but I'll be at a computer reading this page/contributing to my section. Depending on how busy I am, I should be able to get some significant writing in before 4pm today on my section and any additional sections required. RP

- I wont be there either. that does not mean i wont/cant contribute. I'll be on msn or you can just email me. -kirill

=Group members=
Patrick Young [mailto:Rannath@gmail.com Rannath@gmail.com]

Daniel Beimers [mailto:demongyro@gmail.com demongyro@gmail.com]

Andrew Bown [mailto:abown2@connect.carleton.ca abown2@connect.carleton.ca]

Kirill Kashigin [mailto:k.kashigin@gmail.com k.kashigin@gmail.com]

Rovic Perdon [mailto:rperdon@gmail.com rperdon@gmail.com]

Daniel Sont [mailto:dan.sont@gmail.com dan.sont@gmail.com]

=Methodology=
We should probably have our work verified by at least one group member before posting to the actual page

=To Do=
*contribution (conclusion mostly): just need to re-work some sections and follow the cues I left in the Conclusion section.
*critique (conclusion mostly): critique the conclusion of the essay
*style: the style section is largely untouched. Daniel and I([[Rannath]]) have puts some thoughts there, but that section needs to be made into sentences.

=Essay=
==Paper - DONE!!!==
This paper was authored by - Silas Boyd-Wickizer, Austin T. Clements, Yandong Mao, Aleksey Pesterev, M. Frans Kaashoek, Robert Morris, and Nickolai Zeldovich.

They all work at MIT CSAIL.

The paper: [http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf An Analysis of Linux Scalability to Many Cores]

==Background Concepts - DONE!!!==
===memcached: ''Section 3.2''===
memcached is an in-memory hash table server. One instance of memcached running on many different cores is bottlenecked by an internal lock, which is avoided by the MIT team by running one instance per core. Clients each connect to a single instance of memcached, allowing the server to simulate parallelism without needing to make major changes to the application or kernel. With few requests, memcached spends 80% of its time in the kernel on one core, mostly processing packets.[1]

===Apache: ''Section 3.3''===
Apache is a web server that has been used in previous Linux scalability studies. In the case of this study, Apache has been configured to run a separate process on each core. Each process, in turn, has multiple threads (making it a perfect example of parallel programming). Each process uses one of their threads to accepting incoming connections and others are used to process these connections. On a single core processor, Apache spends 60% of its execution time in the kernel.[1]

===gmake: ''Section 3.5''===
gmake is an unofficial default benchmark in the Linux community which is used in this paper to build the Linux kernel. gmake takes a file called a makefile and processes its recipes for the requisite files to determine how and when to remake or recompile code. With a simple command -j or --jobs, gmake can process many of these recipes in parallel. Since gmake creates more processes than cores, it can make proper use of multiple cores to process the recipes.[2] Since gmake involves much reading and writing, in order to prevent bottlenecks due to the filesystem or hardware, the test cases use an in-memory filesystem tmpfs, which gives them a backdoor around the bottlenecks for testing purposes. In addition to this, gmake is limited in scalability due to the serial processes that run at the beginning and end of its execution, which limits its scalability to a small degree. gmake spends much of its execution time with its compiler, processing the recipes and recompiling code, but still spend 7.6% of its time in system time.[1]

[2] http://www.gnu.org/software/make/manual/make.html

==Research problem - DONE!!!==
As technology progresses, the number of core a main processor can have is increasing at an impressive rate. Soon personal computers will have so many cores that scalability will be an issue. There has to be a way that standard user level Linux kernel will scale with a 48-core system[[#Foot1|1]]. The problem with a standard Linux OS is that they are not designed for massive scalability, which will soon prove to be a problem. The issue with scalability is that a solo core will perform much more work compared to a single core working with 47 other cores. Although traditional logic states that the situation makes sense because there are 48 cores dividing the work, the information should be processed as fast as possible with each core doing as much work as possible.

To fix those scalability issues, it is necessary to focus on three major areas: the Linux kernel, user level design and how applications use kernel services. The Linux kernel can be improved by optimizing sharing and use the current advantages of recent improvement to scalability features. On the user level design, applications can be improved so that there is more focus on parallelism since some programs have not implemented those improved features. The final aspect of improving scalability is how an application uses kernel services to better share resources so that different aspects of the program are not conflicting over the same services. All of the bottlenecks are found easily and actually only take simple changes to correct or avoid.[[#Foot1|1]]

This research uses a foundation of previous research discovered during the development of scalability in UNIX systems. The major developments from shared memory machines[[#Foot2|2]] and wait-free synchronization to fast message passing ended up creating a base set of techniques, which can be used to improve scalability. These techniques have been incorporated in all major operating system including Linux, Mac OS X and Windows. Linux has been improved with kernel subsystems, such as Read-Copy-Update, which is an algorithm that is used to avoid locks and atomic instructions which affect scalability.[[#Foot3|3]] There is an excellent base of research on Linux scalability studies that have already been written, on which this research paper can model its testing standards. These papers include research on improving scalability on a 32-core machine.[[#Foot4|4]] In addition, the base of studies can be used to improve the results of these experiments by learning from the previous results. This research may also aid in identifying bottlenecks which speed up creating solutions for those problems.

==Contribution==
All contributions in this paper are the result of the identification and removal or marginalization of bottlenecks.

===What hinders scalability: ''Section 4.1''===
*The percentage of serialization in a program has a lot to do with how much an application can be sped up. This is Amdahl's Law
** Amdahl's Law states that a parallel program can only be sped up by the inverse of the proportion of the program that cannot be made parallel (e.g. 25%(.25) non-parallel --> limit of 4x speedup) (I can't get this to sound right someone fix it please -[[Rannath]] <- I will fix [[Daniel B.]]
*Types of serializing interactions found in the MOSBENCH apps:
**Locking of shared data structure as the number of cores increase leads to an increase in lock wait time
**Writing to shared memory as the number of cores increase leads to an increase in the execution time of the cache coherence protocol
**Competing for space in shared hardware cache as the number of cores increase leads to an increase in cache miss rate
**Competing for shared hardware resources as the number of cores increase leads to time lost waiting for resources
**Not enough tasks for cores leads to idle cores

===Multicore packet processing: ''Section 4.2''===
Linux packet processing technique requires the packets to travel along several queues before it finally becomes available for the application to use. This technique works well for most general socket applications. In recent kernel releases Linux takes advantage of multiple hardware queues (when available on the given network interface) or Receive Packet Steering[1] to direct packet flow onto different cores for processing. Or even go as far as directing packet flow to the core on which the application is running using Receive Flow Steering[2] for even better performance. Linux also attempts to increase performance using a sampling technique where it checks every 20th outgoing packet and directs flow based on its hash. This poses a problem for short lived connections like those associated with Apache since there is great potential for packets to be misdirected.

In general this technique performs poorly when there are numerous open connections spread across multiple cores due to mutex (mutual exclusion) delays and cache misses. In such scenarios its better to process all connections, with associated packets and queues, on one core to avoid said issues. The patched kernel's implementation proposed in this article uses multiple hardware queues (which can be accomplished through Receive Packet Sharing) to direct all packets from a given connection to the same core. In turn Apache is modified to only accept connections if the thread dedicated to processing them is on the same core. If the current core's queue is found to be empty it will attempt to obtain work from queues located on different cores. This configuration is ideal for numerous short connections as all the work for them in accomplished quickly on one core avoiding unnecessary mutex delays associated with packet queues and inter-core cache misses.

[1] J. Corbet. Receive Packet Steering, November 2009. http://lwn.net/Articles/362339/.

[2] J. Edge. Receive Flow Steering, April 2010. http://lwn.net/Articles/382428/.

===Sloppy counters: ''Section 4.3''===
Bottlenecks were encountered when the applications undergoing testing were referencing and updating shared counters for multiple cores. The solution in the paper is to use sloppy counters, which gets each core to track its own separate counts of references and uses a central shared counter to keep all counts on track. This is ideal because each core updates its counts by modifying its per-core counter, usually only needing access to its own local cache, cutting down on waiting for locks or serialization. Sloppy counters are also backwards-compatible with existing shared-counter code, making its implementation much easier to accomplish. The main disadvantages of the sloppy counters are that in situations where object de-allocation occurs often, because the de-allocation itself is an expensive operation, and the counters use up space proportional to the number of cores.

===Lock-free comparison: ''Section 4.4''===
The traditional Linux kernel has very low scalability for name lookups in the directory entry cache. This means there is reduced performance in returning information pertaining to a specific file path when there are multiple threads trying to access files in common parent directories due to the kernel serializing the process. This problem is solved in the patched kernel by introducing a new counter to keep track of threads actively looking at the directory entry cache. If a certain thread threatens an entry currently in use by another, the default locking protocol is used to avoid race conditions. If the activities have no bearing on each other the situation is rightfully ignored allowing for much faster access to different different entries in the directory entry cache.

===Per-Core Data Structures: ''Section 4.5''===
Three centralized data structures were causing bottlenecks - a per-superblock list of open files, vfsmount table, the packet buffers free list. Each data structure was decentralized to per-core versions of itself. In the case of vfsmount the central data structure was maintained, and any per-core misses got written from the central table to the per-core table.

===Eliminating false sharing: ''Section 4.6''===
Misplaced variables on the cache cause different cores to request the same line to be read and written at the same time often enough to significantly impact performance. By moving the often written variable to another line the bottleneck was removed.

===Avoiding unnecessary locking: ''Section 4.7''===
Many locks/mutexes have special cases where they don't need to lock. Likewise mutexes can be split from locking the whole data structure to locking a part of it. Both these changes remove or reduce bottlenecks.

===memcached: ''Section 5.3''===
After improvements to the Linux kernel memcached is Limited by hardware. Improvements to hardware scalability will allow further improvements to memcached.

===Apache: ''Section 5.4''===
Apache like memcached is limited by hardware. At higher core count the network card simply cannot handle the number of packets, the FIFO queue over flows.

===gmake: ''Section 5.6''===

===Conclusion: ''Sections 6 & 7''===
The contribution of this paper is a lot of research that has focus upon techniques and methods for scalability. This is accomplished through programming of applications alongside kernel programming. This research contributes by evaluating the scalability discrepancies of applications programming and kernel programming. Key discoveries in this research show the effectiveness of the kernel in handling scaling amongst CPU cores. In looking at the issue of scalability it is important to note the causes of the factors which hinder scalability.

It has been shown that simple scaling techniques can be effective in increasing scalability. The authors looked at three different approaches to removing the bottlenecks within the system. The first was to see if there were issues within the linux kernel application, the second was to identify issues with the application design and the third was to address how the application interacts with the linux kernel services. Through this approach, the authors were able to quickly identify problems such as bottlenecks and apply simple techniques in fixing the issues at hand to reap some beneficial aspects. Some of the sections listed provide insight into the improvements that can be reaped from these optimizations.

Through this research on various techniques as listed above, it was determined by the authors that the Linux kernel itself has many incorporated techniques used to improve scalability. The authors actually go on to speculate that "perhaps it is the case that Linux's system-call API is well suited to an implementation that avoids unnecessary contention over kernel objects." This tends to show that the work in the Linux community has improved Linux a large amount and is current with modern techniques for optimization. It could also be interpreted from the paper that it may be to the benefit of the community to change how the applications are programmed rather than make changes to the Linux kernel in order to make scalability improvements. This may indicate that what has come before is done quite well when considering that the Linux kernel optimizations showed more improvement when put in conjunction with the application improvements.

==Critique==
What is good and not-so-good about this paper? You may discuss both the style and content;
be sure to ground your discussion with specific references. Simple assertions that something is good or bad is not enough - you must explain why.
Since this is a "my implementation is better then your implementation" paper the "goodness" of content can be impartially determined by its fairness and the honesty of the authors.

===Content(Fairness): ''Section 5''===

====memcached: ''Section 5.3''====
memcached is treated with near perfect fairness in the paper. Its an in-memory service, so the ignored storage IO bottleneck does not affect it at all. Likewise the "stock" and "PK" implementations are given the same test suite, so there is no advantage given to either. memcached itself is non-scalable, so the MIT team was forced to run one instance per-core to keep up throughput. The FAQ at memcached.org's wiki suggests using multiple implementations per-server as a work around to another problem, which implies that running multiple instances of the server is the same, or nearly the same, as running one larger server [3]. In the end memcached was bottlenecked by the network card.

====Apache: ''Section 5.4''====
Linux has a built in kernel flaw where network packets are forced to travel though multiple queues before they arrive at queue where they can be processed by the application. This imposes significant costs on multi-core systems due to queue locking costs. This flaw inherently diminishes the performance of Apache on multi-core system due to multiple threads spread across cores being forced to deal with these mutex (mutual exclusion) costs. For the sake of this experiment Apache had a separate instance on every core listening on different ports which is not a practical real world application but merely an attempt to implement better parallel execution on a traditional kernel. The patched kernel implementation of the network stack is also specific to the problem at hand, which is processing multiple short lived connections across multiple cores. Although this provides a performance increase in the given scenario, in more general applications network performance might suffer. These tests were also rigged to avoid bottlenecks in place by network and file storage hardware. Meaning, making the proposed modifications to the kernel wont necessarily produce the same increase in productivity as described in the article. This is very much evident in the test where performance degrades past 36 cores due to limitation of the networking hardware.

''Which is not a problem as the paper specifically states that they are testing what they can improve in spite of hardware limitation.'' - [[Rannath]]

====gmake: ''Section 5.6''====
Since the inherent nature of gmake makes it quite parallel, the testing and updating that were attempted on gmake resulted in essentially the same scalability results for both the stock and modified kernel. The only change that was found was that gmake spent slightly less time at the system level because of the changes that were made to the system's caching. As stated in the paper, the execution time of gmake relies quite heavily on the compiler that is uses with gmake, so depending on which compiler was chosen, gmake could run worse or even slightly better. In any case, there seems to be no fairness concerns when it comes to the scalability testing of gmake as the same application load-out was used for all of the tests.

====Conclusion: ''Sections 6 & 7''====
Given that all tests are more or less fair for the purposes of the benchmarks. They would support the Hypothesis that Linux can be made to scale, at least to 48 cores. Thus the conclusion is fair iff the rest of the paper is fair.

Now you just have to fill in how fair the rest of the paper is.

===Style===
Style Criterion (feel free to add I have no idea what should go here):
- does the paper present information out of order?
- does the paper present needless information?
- does the paper have any sections that are inherently confusing? Wrong?
- is the paper easy to read through, or does it change subjects repeatedly?
- does the paper have too many "long-winded" sentences, making it seem like the authors are just trying to add extra words to make it seem more important? - I think maybe limit this to run-on sentences.
- Check for grammar

Everything seems to be in logical order. I couldn't find any needless info. Nothing inherently confusing or wrong. Nothing bad on the grammar front either. - Rannath

Some acronyms aren't explained before they are used, so some people reading the paper may get confused as to what they mean (e.g. Linux TLB). Since this paper is meant to be formal, acronyms should be explained, with some exceptions like OS and IBM. - Daniel B.

Your example has no impact on the paper, it was in the "look here for more info" section. Most people wouldn't know what a "translation look-aside buffer" is either.

==References==

[3] memcached's wiki: http://code.google.com/p/memcached/wiki/FAQ#Can_I_use_different_size_caches_across_servers_and_will_memcache

You will almost certainly have to refer to other resources; please cite these resources in the style of citation of the papers assigned (inlined numbered references). Place your bibliographic entries in this section.

'''the paper itself doesn't need to be referenced more than once as this is a critique of the paper...'''
[1] Silas Boyd-Wickizer et al. "An Analysis of Linux Scalability to Many Cores". In ''OSDI '10, 9th USENIX Symposium on OS Design and Implementation'', Vancouver, BC, Canada, 2010. http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf.

gmake:

[http://www.gnu.org/software/make/manual/make.html gmake Manual]

[http://www.gnu.org/software/make/ gmake Main Page]

==Deprecated==
===Background Concepts===
* Exim: ''Section 3.1'':
**Exim is a mail server for Unix. It's fairly parallel. The server forks a new process for each connection and twice to deliver each message. It spends 69% of its time in the kernel on a single core.

* PostgreSQL: ''Section 3.4'':
**As implied by the name PostgreSQL is a SQL database. PostgreSQL starts a separate process for each connection and uses kernel locking interfaces extensively to provide concurrent access to the database. Due to bottlenecks introduced in its code and in the kernel code, the amount of time PostgreSQL spends in the kernel increases very rapidly with addition of new cores. On a single core system PostgreSQL spends only 1.5% of its time in the kernel. On a 48 core system the execution time in the kernel jumps to 82%.

Talk:COMP 3000 Essay 2 2010 Question 1

2010-12-02T17:52:14Z

Kkashigi: /* Multicore packet processing: Section 4.2 */

=Class and Notices=
(Nov. 30, 2010) Prof. Anil stated that we should focus on the 3 easiest to understand parts in section 5 and elaborate on them.

- Also, I, Daniel B., work Thursday night, so I will be finishing up as much of my part as I can for the essay before Thursday morning's class, then maybe we can all meet up in a lab in HP and put the finishing touches on the essay. I will be available online Wednesday night from about 6:30pm onwards and will be in the game dev lab or CCSS lounge Wednesday morning from about 11am to 2pm if anyone would like to meet up with me at those times.

- [[I suggest we meet up Thursday morning after Operating Systems in order to discuss and finalize the essay. Maybe we can even designate a lab for the group to meet up in. Any suggestions?]] - Daniel B.

- HP 3115 since there wont be a class in there (as its our tutorial and we know there won't be anyone there)
-- Go to Wireless Lab next to CCSS Lounge. Andrew and Dan B. will be there.

- If its all the same to you guys mind if I just join you via msn or iirc? Or phone if you really want. -Rannath

- I'm working today, but I'll be at a computer reading this page/contributing to my section. Depending on how busy I am, I should be able to get some significant writing in before 4pm today on my section and any additional sections required. RP

- I wont be there either. that does not mean i wont/cant contribute. I'll be on msn or you can just email me. -kirill

=Group members=
Patrick Young [mailto:Rannath@gmail.com Rannath@gmail.com]

Daniel Beimers [mailto:demongyro@gmail.com demongyro@gmail.com]

Andrew Bown [mailto:abown2@connect.carleton.ca abown2@connect.carleton.ca]

Kirill Kashigin [mailto:k.kashigin@gmail.com k.kashigin@gmail.com]

Rovic Perdon [mailto:rperdon@gmail.com rperdon@gmail.com]

Daniel Sont [mailto:dan.sont@gmail.com dan.sont@gmail.com]

=Methodology=
We should probably have our work verified by at least one group member before posting to the actual page

=To Do=
*Improve the grammar/structure of the paper section & add links to supplementary info
*flesh out the whole lot

===Claim Sections===
* So here is the claims and unclaimed section. Add your name next to one if you want to take it on.
** gmake - Daniel B.
** memcached - Rannath
** Apache - Kirill
** [[(Exim, PostgreSQL, Metis, and Psearchy will not be needed as the professor said we only need to explain 3)]]
** Research Problem - Andrew
** Contribution - Rovic
** Essay Conclusion (also discussion) - Everyone
** Critic, Style - Everyone
** References - Everyone

=Essay=
==Paper - DONE!!!==
This paper was authored by - Silas Boyd-Wickizer, Austin T. Clements, Yandong Mao, Aleksey Pesterev, M. Frans Kaashoek, Robert Morris, and Nickolai Zeldovich.

They all work at MIT CSAIL.

The paper: [http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf An Analysis of Linux Scalability to Many Cores]

==Background Concepts - DONE!!!==
===memcached: ''Section 3.2''===
memcached is an in-memory hash table server. One instance of memcached running on many different cores is bottlenecked by an internal lock, which is avoided by the MIT team by running one instance per core. Clients each connect to a single instance of memcached, allowing the server to simulate parallelism without needing to make major changes to the application or kernel. With few requests, memcached spends 80% of its time in the kernel on one core, mostly processing packets.[1]

===Apache: ''Section 3.3''===
Apache is a web server that has been used in previous Linux scalability studies. In the case of this study, Apache has been configured to run a separate process on each core. Each process, in turn, has multiple threads (making it a perfect example of parallel programming). Each process uses one of their threads to accepting incoming connections and others are used to process these connections. On a single core processor, Apache spends 60% of its execution time in the kernel.[1]

===gmake: ''Section 3.5''===
gmake is an unofficial default benchmark in the Linux community which is used in this paper to build the Linux kernel. gmake takes a file called a makefile and processes its recipes for the requisite files to determine how and when to remake or recompile code. With a simple command -j or --jobs, gmake can process many of these recipes in parallel. Since gmake creates more processes than cores, it can make proper use of multiple cores to process the recipes.[2] Since gmake involves much reading and writing, in order to prevent bottlenecks due to the filesystem or hardware, the test cases use an in-memory filesystem tmpfs, which gives them a backdoor around the bottlenecks for testing purposes. In addition to this, gmake is limited in scalability due to the serial processes that run at the beginning and end of its execution, which limits its scalability to a small degree. gmake spends much of its execution time with its compiler, processing the recipes and recompiling code, but still spend 7.6% of its time in system time.[1]

[2] http://www.gnu.org/software/make/manual/make.html

==Research problem==
my references are just below because it is easier for numbering the data later.

As technological progress the number of core a main processor can have is increasing at an impressive rate. Soon personal computers will have so many cores that scalability will be an issue. There has to be a way that standard user level Linux kernel will scale with a 48-core system[1]. The problem with a standard Linux operating is they are not designed for massive scalability which will soon be a problem. The issue with scalability is that a solo core will perform much more work compared to a single core working with 47 other cores. Although traditional logic that situation makes sense because 48 cores are dividing the work. But when processing information a process the main goal is to finish so as long as possible every core should be doing a much work as possible.

To fix those scalability issues it is necessary to focus on three major areas: the Linux kernel, user level design and how application use of kernel services. The Linux kernel can be improved be to improve sharing and have the advantage of recent iterations are beginning to implement scalability features. On the user level design applications can be improved so that there is more focus on parallelism since some programs have not implements those improved features. The final aspect of improving scalability is how an application uses kernel services to share resources better so that different aspects of the program are not conflicting over the same services. All of the bottlenecks are found actually only take a little work to avoid.[1]

This research is based on much research which was created before in the development of scalability for UNIX system. The major developments from shared memory machines [2], wait-free synchronization to fast message passing have created a base set of techniques which can be used to improve scalability. These techniques have been incorporated in all major operation system including Linux, Mac OS X and Windows. Linux has been improved with kernel subsystems such as Read-Copy-Update which an algorithm for which is used to avoid locks and atomic instructions which lower scalability.[3] The is also an excellent base a research on Linux scalability studies to base this research paper. These paper include a on doing scalability on a 32-core machine. [4] That research can improve the results by learning from the experiments already performed by researchers. This research also aid identifying bottlenecks which speed up researching solutions for those bottlenecks.

[2] J. Kuskin, D. Ofelt, M. Heinrich, J. Heinlein, R. Simoni, K. Gharachorloo, J. Chapin, D. Nakahira, J. Baxter, M. Horowitz, A. Gupta, M. Rosenblum, and J. Hennessy. The Stanford FLASH multiprocessor. In Proc. of the 21st ISCA, pages 302–313,1994.

[3] P. E. McKenney, D. Sarma, A. Arcangeli, A. Kleen, O. Krieger, and R. Russell. Read-copy-update. In Proceedings of the Linux Symposium 2002, pages 338-367, Ottawa Ontario, June 2002

[4] C. Yan, Y. Chen, and S. Yuanchun. OSMark: A benchmark suite for understanding parallel scalability of operating systems on large scale multi-cores. In 2009 2nd International Conference on Computer Science and Information Technology, pages 313–317, 2009

==Contribution==
What was implemented? Why is it any better than what came before?

Summarize info from Section 4.2 onwards, maybe put graphs from Section 5 here to provide support for improvements (if that isn't unethical/illegal)?

- So long as we cite the paper and don't pretend the graphs are ours, we are ok, since we are writing an explanation/critic of the paper.

All contributions in this paper are the result of the identification and removal or marginalization of bottlenecks.

===What hinders scalability: ''Section 4.1''===
*The percentage of serialization in a program has a lot to do with how much an application can be sped up. This is Amdahl's Law
** Amdahl's Law states that a parallel program can only be sped up by the inverse of the proportion of the program that cannot be made parallel (e.g. 25%(.25) non-parallel --> limit of 4x speedup) (I can't get this to sound right someone fix it please -[[Rannath]] <- I will fix [[Daniel B.]]
*Types of serializing interactions found in the MOSBENCH apps:
**Locking of shared data structure as the number of cores increase leads to an increase in lock wait time
**Writing to shared memory as the number of cores increase leads to an increase in the execution time of the cache coherence protocol
**Competing for space in shared hardware cache as the number of cores increase leads to an increase in cache miss rate
**Competing for shared hardware resources as the number of cores increase leads to time lost waiting for resources
**Not enough tasks for cores leads to idle cores

===Multicore packet processing: ''Section 4.2''===
Linux packet processing technique requires the packets to travel along several queues before it finally becomes available for the application to use. This technique works well for most general socket applications. In recent kernel releases Linux takes advantage of multiple hardware queues (when available on the given network interface) or Receive Packet Steering[1] to direct packet flow onto different cores for processing. Or even go as far as directing packet flow to the core on which the application is running using Receive Flow Steering[2] for even better performance. Linux also attempts to increase performance using a sampling technique where it checks every 20th outgoing packet and directs flow based on its hash. This poses a problem for short lived connections like those associated with Apache since there is great potential for packets to be misdirected.

In general this technique performs poorly when there are numerous open connections spread across multiple cores due to mutex (mutual exclusion) delays and cache misses. In such scenarios its better to process all connections, with associated packets and queues, on one core to avoid said issues. The patched kernel's implementation proposed in this article uses multiple hardware queues (which can be accomplished through Receive Packet Sharing) to direct all packets from a given connection to the same core. In turn Apache is modified to only accept connections if the thread dedicated to processing them is on the same core. If the current core's queue is found to be empty it will attempt to obtain work from queues located on different cores. This configuration is ideal for numerous short connections as all the work for them in accomplished quickly on one core avoiding unnecessary mutex delays associated with packet queues and inter-core cache misses.

[1] J. Corbet. Receive Packet Steering, November 2009. http://lwn.net/Articles/362339/.

[2] J. Edge. Receive Flow Steering, April 2010. http://lwn.net/Articles/382428/.

===Sloppy counters: ''Section 4.3''===
Bottlenecks were encountered when the applications undergoing testing were referencing and updating shared counters for multiple cores. The solution in the paper is to use sloppy counters, which gets each core to track its own separate counts of references and uses a central shared counter to keep all counts on track. This is ideal because each core updates its counts by modifying its per-core counter, usually only needing access to its own local cache, cutting down on waiting for locks or serialization. Sloppy counters are also backwards-compatible with existing shared-counter code, making its implementation much easier to accomplish. The main disadvantages of the sloppy counters are that in situations where object de-allocation occurs often, because the de-allocation itself is an expensive operation, and the counters use up space proportional to the number of cores.

===Lock-free comparison: ''Section 4.4''===
This section describes a specific instance of unnecessary locking.

===Per-Core Data Structures: ''Section 4.5''===
Three centralized data structures were causing bottlenecks - a per-superblock list of open files, vfsmount table, the packet buffers free list. Each data structure was decentralized to per-core versions of itself. In the case of vfsmount the central data structure was maintained, and any per-core misses got written from the central table to the per-core table.

===Eliminating false sharing: ''Section 4.6''===
Misplaced variables on the cache cause different cores to request the same line to be read and written at the same time often enough to significantly impact performance. By moving the often written variable to another line the bottleneck was removed.

===Avoiding unnecessary locking: ''Section 4.7''===
Many locks/mutexes have special cases where they don't need to lock. Likewise mutexes can be split from locking the whole data structure to locking a part of it. Both these changes remove or reduce bottlenecks.

==Conclusion: ''Sections 6 & 7''==
===Work in Progress===

====[[Rovic P.]]====
-I'm just using this as a notepad, do not copy/paste this section, I will put in a properly written set of paragraphs which will fit with the contribution questions asked. -RP

This research contributes by evaluating the scalability discrepancies of applications programming and kernel programming. Key discoveries in this research show the effectiveness of the kernel in handling scaling amongst CPU cores. This has also shown that scaling in application programming should be more the focus. It has been shown that simple scaling techniques (list techniques) such as programming parallelism (look up more stuff to back this up and quotes). (Sloppy counter effectiveness, possible positive contributions, what has been used (internet search), what hasn’t been used.) Read conclusion, 2nd paragraph.

One reason the
required changes are modest is that stock Linux already
incorporates many modifications to improve scalability.
More speculatively, perhaps it is the case that Linux’s
system-call API is well suited to an implementation that
avoids unnecessary contention over kernel objects.

====[[Rannath]]====
Everything so far indicates that the MOSBENCH applications can scale to 48 cores. This scaling required a few modest changes to remove bottlenecks. The MIT team speculate that that trend will continue as the number of cores increase. They also state that things not bottlenecked by the CPU are harder to fix.

We can eliminate most kernel bottlenecks that the applications hits most often with minor changes. Most changes were well known methodology, with the exception of Sloppy counters. This study is limited by the removal of the IO bottleneck, but it does suggest that traditional implementations can be made scalable.

==Critique==
What is good and not-so-good about this paper? You may discuss both the style and content;
be sure to ground your discussion with specific references. Simple assertions that something is good or bad is not enough - you must explain why.
Since this is a "my implementation is better then your implementation" paper the "goodness" of content can be impartially determined by its fairness and the honesty of the authors.

===Content(Fairness): ''Section 5''===

====memcached: ''Section 5.3''====
memcached is treated with near perfect fairness in the paper. Its an in-memory service, so the ignored storage IO bottleneck does not affect it at all. Likewise the "stock" and "PK" implementations are given the same test suite, so there is no advantage given to either. memcached itself is non-scalable, so the MIT team was forced to run one instance per-core to keep up throughput. The FAQ at memcached.org's wiki suggests using multiple implementations per-server as a work around to another problem, which implies that running multiple instances of the server is the same, or nearly the same, as running one larger server [3]. In the end memcached was bottlenecked by the network card.

====Apache: ''Section 5.4''====
Linux has a built in kernel flaw where network packets are forced to travel though multiple queues before they arrive at queue where they can be processed by the application. This imposes significant costs on multi-core systems due to queue locking costs. This flaw inherently diminishes the performance of Apache on multi-core system due to multiple threads spread across cores being forced to deal with these mutex (mutual exclusion) costs. For the sake of this experiment Apache had a separate instance on every core listening on different ports which is not a practical real world application but merely an attempt to implement better parallel execution on a traditional kernel. The patched kernel implementation of the network stack is also specific to the problem at hand, which is processing multiple short lived connections across multiple cores. Although this provides a performance increase in the given scenario, in more general applications network performance might suffer. These tests were also rigged to avoid bottlenecks in place by network and file storage hardware. Meaning, making the proposed modifications to the kernel wont necessarily produce the same increase in productivity as described in the article. This is very much evident in the test where performance degrades past 36 cores due to limitation of the networking hardware.

''Which is not a problem as the paper specifically states that they are testing what they can improve in spite of hardware limitation.'' - [[Rannath]]

====gmake: ''Section 5.6''====
Since the inherent nature of gmake makes it quite parallel, the testing and updating that were attempted on gmake resulted in essentially the same scalability results for both the stock and modified kernel. The only change that was found was that gmake spent slightly less time at the system level because of the changes that were made to the system's caching. As stated in the paper, the execution time of gmake relies quite heavily on the compiler that is uses with gmake, so depending on which compiler was chosen, gmake could run worse or even slightly better. In any case, there seems to be no fairness concerns when it comes to the scalability testing of gmake as the same application load-out was used for all of the tests.

====Conclusion: ''Sections 6 & 7''====
Given that all tests are more or less fair for the purposes of the benchmarks. They would support the Hypothesis that Linux can be made to scale, at least to 48 cores. Thus the conclusion is fair iff the rest of the paper is fair.

Now you just have to fill in how fair the rest of the paper is.

===Style===
Style Criterion (feel free to add I have no idea what should go here):
- does the paper present information out of order?
- does the paper present needless information?
- does the paper have any sections that are inherently confusing? Wrong?
- is the paper easy to read through, or does it change subjects repeatedly?
- does the paper have too many "long-winded" sentences, making it seem like the authors are just trying to add extra words to make it seem more important? - I think maybe limit this to run-on sentences.
- Check for grammar

Everything seems to be in logical order. I couldn't find any needless info. Nothing inherently confusing or wrong. Nothing bad on the grammar front either. - Rannath

Some acronyms aren't explained before they are used, so some people reading the paper may get confused as to what they mean (e.g. Linux TLB). Since this paper is meant to be formal, acronyms should be explained, with some exceptions like OS and IBM. - Daniel B.

Your example has no impact on the paper, it was in the "look here for more info" section. Most people wouldn't know what a "translation look-aside buffer" is either.

==References==

[3] memcached's wiki: http://code.google.com/p/memcached/wiki/FAQ#Can_I_use_different_size_caches_across_servers_and_will_memcache

You will almost certainly have to refer to other resources; please cite these resources in the style of citation of the papers assigned (inlined numbered references). Place your bibliographic entries in this section.

'''the paper itself doesn't need to be referenced more than once as this is a critique of the paper...'''
[1] Silas Boyd-Wickizer et al. "An Analysis of Linux Scalability to Many Cores". In ''OSDI '10, 9th USENIX Symposium on OS Design and Implementation'', Vancouver, BC, Canada, 2010. http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf.

gmake:

[http://www.gnu.org/software/make/manual/make.html gmake Manual]

[http://www.gnu.org/software/make/ gmake Main Page]

==Deprecated==
===Background Concepts===
* Exim: ''Section 3.1'':
**Exim is a mail server for Unix. It's fairly parallel. The server forks a new process for each connection and twice to deliver each message. It spends 69% of its time in the kernel on a single core.

* PostgreSQL: ''Section 3.4'':
**As implied by the name PostgreSQL is a SQL database. PostgreSQL starts a separate process for each connection and uses kernel locking interfaces extensively to provide concurrent access to the database. Due to bottlenecks introduced in its code and in the kernel code, the amount of time PostgreSQL spends in the kernel increases very rapidly with addition of new cores. On a single core system PostgreSQL spends only 1.5% of its time in the kernel. On a 48 core system the execution time in the kernel jumps to 82%.

Talk:COMP 3000 Essay 2 2010 Question 1

2010-12-02T17:48:27Z

Kkashigi: /* Multicore packet processing: Section 4.2 */

=Class and Notices=
(Nov. 30, 2010) Prof. Anil stated that we should focus on the 3 easiest to understand parts in section 5 and elaborate on them.

- Also, I, Daniel B., work Thursday night, so I will be finishing up as much of my part as I can for the essay before Thursday morning's class, then maybe we can all meet up in a lab in HP and put the finishing touches on the essay. I will be available online Wednesday night from about 6:30pm onwards and will be in the game dev lab or CCSS lounge Wednesday morning from about 11am to 2pm if anyone would like to meet up with me at those times.

- [[I suggest we meet up Thursday morning after Operating Systems in order to discuss and finalize the essay. Maybe we can even designate a lab for the group to meet up in. Any suggestions?]] - Daniel B.

- HP 3115 since there wont be a class in there (as its our tutorial and we know there won't be anyone there)
-- Go to Wireless Lab next to CCSS Lounge. Andrew and Dan B. will be there.

- If its all the same to you guys mind if I just join you via msn or iirc? Or phone if you really want. -Rannath

- I'm working today, but I'll be at a computer reading this page/contributing to my section. Depending on how busy I am, I should be able to get some significant writing in before 4pm today on my section and any additional sections required. RP

- I wont be there either. that does not mean i wont/cant contribute. I'll be on msn or you can just email me. -kirill

=Group members=
Patrick Young [mailto:Rannath@gmail.com Rannath@gmail.com]

Daniel Beimers [mailto:demongyro@gmail.com demongyro@gmail.com]

Andrew Bown [mailto:abown2@connect.carleton.ca abown2@connect.carleton.ca]

Kirill Kashigin [mailto:k.kashigin@gmail.com k.kashigin@gmail.com]

Rovic Perdon [mailto:rperdon@gmail.com rperdon@gmail.com]

Daniel Sont [mailto:dan.sont@gmail.com dan.sont@gmail.com]

=Methodology=
We should probably have our work verified by at least one group member before posting to the actual page

=To Do=
*Improve the grammar/structure of the paper section & add links to supplementary info
*flesh out the whole lot

===Claim Sections===
* So here is the claims and unclaimed section. Add your name next to one if you want to take it on.
** gmake - Daniel B.
** memcached - Rannath
** Apache - Kirill
** [[(Exim, PostgreSQL, Metis, and Psearchy will not be needed as the professor said we only need to explain 3)]]
** Research Problem - Andrew
** Contribution - Rovic
** Essay Conclusion (also discussion) - Everyone
** Critic, Style - Everyone
** References - Everyone

=Essay=
==Paper - DONE!!!==
This paper was authored by - Silas Boyd-Wickizer, Austin T. Clements, Yandong Mao, Aleksey Pesterev, M. Frans Kaashoek, Robert Morris, and Nickolai Zeldovich.

They all work at MIT CSAIL.

The paper: [http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf An Analysis of Linux Scalability to Many Cores]

==Background Concepts - DONE!!!==
===memcached: ''Section 3.2''===
memcached is an in-memory hash table server. One instance of memcached running on many different cores is bottlenecked by an internal lock, which is avoided by the MIT team by running one instance per core. Clients each connect to a single instance of memcached, allowing the server to simulate parallelism without needing to make major changes to the application or kernel. With few requests, memcached spends 80% of its time in the kernel on one core, mostly processing packets.[1]

===Apache: ''Section 3.3''===
Apache is a web server that has been used in previous Linux scalability studies. In the case of this study, Apache has been configured to run a separate process on each core. Each process, in turn, has multiple threads (making it a perfect example of parallel programming). Each process uses one of their threads to accepting incoming connections and others are used to process these connections. On a single core processor, Apache spends 60% of its execution time in the kernel.[1]

===gmake: ''Section 3.5''===
gmake is an unofficial default benchmark in the Linux community which is used in this paper to build the Linux kernel. gmake takes a file called a makefile and processes its recipes for the requisite files to determine how and when to remake or recompile code. With a simple command -j or --jobs, gmake can process many of these recipes in parallel. Since gmake creates more processes than cores, it can make proper use of multiple cores to process the recipes.[2] Since gmake involves much reading and writing, in order to prevent bottlenecks due to the filesystem or hardware, the test cases use an in-memory filesystem tmpfs, which gives them a backdoor around the bottlenecks for testing purposes. In addition to this, gmake is limited in scalability due to the serial processes that run at the beginning and end of its execution, which limits its scalability to a small degree. gmake spends much of its execution time with its compiler, processing the recipes and recompiling code, but still spend 7.6% of its time in system time.[1]

[2] http://www.gnu.org/software/make/manual/make.html

==Research problem==
my references are just below because it is easier for numbering the data later.

As technological progress the number of core a main processor can have is increasing at an impressive rate. Soon personal computers will have so many cores that scalability will be an issue. There has to be a way that standard user level Linux kernel will scale with a 48-core system[1]. The problem with a standard Linux operating is they are not designed for massive scalability which will soon be a problem. The issue with scalability is that a solo core will perform much more work compared to a single core working with 47 other cores. Although traditional logic that situation makes sense because 48 cores are dividing the work. But when processing information a process the main goal is to finish so as long as possible every core should be doing a much work as possible.

To fix those scalability issues it is necessary to focus on three major areas: the Linux kernel, user level design and how application use of kernel services. The Linux kernel can be improved be to improve sharing and have the advantage of recent iterations are beginning to implement scalability features. On the user level design applications can be improved so that there is more focus on parallelism since some programs have not implements those improved features. The final aspect of improving scalability is how an application uses kernel services to share resources better so that different aspects of the program are not conflicting over the same services. All of the bottlenecks are found actually only take a little work to avoid.[1]

This research is based on much research which was created before in the development of scalability for UNIX system. The major developments from shared memory machines [2], wait-free synchronization to fast message passing have created a base set of techniques which can be used to improve scalability. These techniques have been incorporated in all major operation system including Linux, Mac OS X and Windows. Linux has been improved with kernel subsystems such as Read-Copy-Update which an algorithm for which is used to avoid locks and atomic instructions which lower scalability.[3] The is also an excellent base a research on Linux scalability studies to base this research paper. These paper include a on doing scalability on a 32-core machine. [4] That research can improve the results by learning from the experiments already performed by researchers. This research also aid identifying bottlenecks which speed up researching solutions for those bottlenecks.

[2] J. Kuskin, D. Ofelt, M. Heinrich, J. Heinlein, R. Simoni, K. Gharachorloo, J. Chapin, D. Nakahira, J. Baxter, M. Horowitz, A. Gupta, M. Rosenblum, and J. Hennessy. The Stanford FLASH multiprocessor. In Proc. of the 21st ISCA, pages 302–313,1994.

[3] P. E. McKenney, D. Sarma, A. Arcangeli, A. Kleen, O. Krieger, and R. Russell. Read-copy-update. In Proceedings of the Linux Symposium 2002, pages 338-367, Ottawa Ontario, June 2002

[4] C. Yan, Y. Chen, and S. Yuanchun. OSMark: A benchmark suite for understanding parallel scalability of operating systems on large scale multi-cores. In 2009 2nd International Conference on Computer Science and Information Technology, pages 313–317, 2009

==Contribution==
What was implemented? Why is it any better than what came before?

Summarize info from Section 4.2 onwards, maybe put graphs from Section 5 here to provide support for improvements (if that isn't unethical/illegal)?

- So long as we cite the paper and don't pretend the graphs are ours, we are ok, since we are writing an explanation/critic of the paper.

All contributions in this paper are the result of the identification and removal or marginalization of bottlenecks.

===What hinders scalability: ''Section 4.1''===
*The percentage of serialization in a program has a lot to do with how much an application can be sped up. This is Amdahl's Law
** Amdahl's Law states that a parallel program can only be sped up by the inverse of the proportion of the program that cannot be made parallel (e.g. 25%(.25) non-parallel --> limit of 4x speedup) (I can't get this to sound right someone fix it please -[[Rannath]] <- I will fix [[Daniel B.]]
*Types of serializing interactions found in the MOSBENCH apps:
**Locking of shared data structure as the number of cores increase leads to an increase in lock wait time
**Writing to shared memory as the number of cores increase leads to an increase in the execution time of the cache coherence protocol
**Competing for space in shared hardware cache as the number of cores increase leads to an increase in cache miss rate
**Competing for shared hardware resources as the number of cores increase leads to time lost waiting for resources
**Not enough tasks for cores leads to idle cores

===Multicore packet processing: ''Section 4.2''===
Linux packet processing technique requires the packets to travel along several queues before it finally becomes available for the application to use. This technique works well for most general socket applications. In recent kernel releases Linux takes advantage of multiple hardware queues (when available on the given network interface) or Receive Packet Steering<ref>J. Corbet. Receive Packet Steering, November 2009. http://lwn.net/Articles/362339/.</ref> to direct packet flow onto different cores for processing. Or even go as far as directing packet flow to the core on which the application is running using Receive Flow Steering<ref>J. Edge. Receive Flow Steering, April 2010. http://lwn.net/Articles/382428/.</ref> for even better performance. Linux also attempts to increase performance using a sampling technique where it checks every 20th outgoing packet and directs flow based on its hash. This poses a problem for short lived connections like those associated with Apache since there is great potential for packets to be misdirected.

In general this technique performs poorly when there are numerous open connections spread across multiple cores due to mutex (mutual exclusion) delays and cache misses. In such scenarios its better to process all connections, with associated packets and queues, on one core to avoid said issues. The patched kernel's implementation proposed in this article uses multiple hardware queues (which can be accomplished through Receive Packet Sharing) to direct all packets from a given connection to the same core. In turn Apache is modified to only accept connections if the thread dedicated to processing them is on the same core. If the current core's queue is found to be empty it will attempt to obtain work from queues located on different cores. This configuration is ideal for numerous short connections as all the work for them in accomplished quickly on one core avoiding unnecessary mutex delays associated with packet queues and inter-core cache misses.

{{Reflist}}

[1] J. Corbet. Receive Packet Steering, November 2009. http://lwn.net/Articles/362339/.

[2] J. Edge. Receive Flow Steering, April 2010. http://lwn.net/Articles/382428/.

===Sloppy counters: ''Section 4.3''===
Bottlenecks were encountered when the applications undergoing testing were referencing and updating shared counters for multiple cores. The solution in the paper is to use sloppy counters, which gets each core to track its own separate counts of references and uses a central shared counter to keep all counts on track. This is ideal because each core updates its counts by modifying its per-core counter, usually only needing access to its own local cache, cutting down on waiting for locks or serialization. Sloppy counters are also backwards-compatible with existing shared-counter code, making its implementation much easier to accomplish. The main disadvantages of the sloppy counters are that in situations where object de-allocation occurs often, because the de-allocation itself is an expensive operation, and the counters use up space proportional to the number of cores.

===Lock-free comparison: ''Section 4.4''===
This section describes a specific instance of unnecessary locking.

===Per-Core Data Structures: ''Section 4.5''===
Three centralized data structures were causing bottlenecks - a per-superblock list of open files, vfsmount table, the packet buffers free list. Each data structure was decentralized to per-core versions of itself. In the case of vfsmount the central data structure was maintained, and any per-core misses got written from the central table to the per-core table.

===Eliminating false sharing: ''Section 4.6''===
Misplaced variables on the cache cause different cores to request the same line to be read and written at the same time often enough to significantly impact performance. By moving the often written variable to another line the bottleneck was removed.

===Avoiding unnecessary locking: ''Section 4.7''===
Many locks/mutexes have special cases where they don't need to lock. Likewise mutexes can be split from locking the whole data structure to locking a part of it. Both these changes remove or reduce bottlenecks.

==Conclusion: ''Sections 6 & 7''==
===Work in Progress===

====[[Rovic P.]]====
-I'm just using this as a notepad, do not copy/paste this section, I will put in a properly written set of paragraphs which will fit with the contribution questions asked. -RP

This research contributes by evaluating the scalability discrepancies of applications programming and kernel programming. Key discoveries in this research show the effectiveness of the kernel in handling scaling amongst CPU cores. This has also shown that scaling in application programming should be more the focus. It has been shown that simple scaling techniques (list techniques) such as programming parallelism (look up more stuff to back this up and quotes). (Sloppy counter effectiveness, possible positive contributions, what has been used (internet search), what hasn’t been used.) Read conclusion, 2nd paragraph.

One reason the
required changes are modest is that stock Linux already
incorporates many modifications to improve scalability.
More speculatively, perhaps it is the case that Linux’s
system-call API is well suited to an implementation that
avoids unnecessary contention over kernel objects.

====[[Rannath]]====
Everything so far indicates that the MOSBENCH applications can scale to 48 cores. This scaling required a few modest changes to remove bottlenecks. The MIT team speculate that that trend will continue as the number of cores increase. They also state that things not bottlenecked by the CPU are harder to fix.

We can eliminate most kernel bottlenecks that the applications hits most often with minor changes. Most changes were well known methodology, with the exception of Sloppy counters. This study is limited by the removal of the IO bottleneck, but it does suggest that traditional implementations can be made scalable.

==Critique==
What is good and not-so-good about this paper? You may discuss both the style and content;
be sure to ground your discussion with specific references. Simple assertions that something is good or bad is not enough - you must explain why.
Since this is a "my implementation is better then your implementation" paper the "goodness" of content can be impartially determined by its fairness and the honesty of the authors.

===Content(Fairness): ''Section 5''===

====memcached: ''Section 5.3''====
memcached is treated with near perfect fairness in the paper. Its an in-memory service, so the ignored storage IO bottleneck does not affect it at all. Likewise the "stock" and "PK" implementations are given the same test suite, so there is no advantage given to either. memcached itself is non-scalable, so the MIT team was forced to run one instance per-core to keep up throughput. The FAQ at memcached.org's wiki suggests using multiple implementations per-server as a work around to another problem, which implies that running multiple instances of the server is the same, or nearly the same, as running one larger server [3]. In the end memcached was bottlenecked by the network card.

====Apache: ''Section 5.4''====
Linux has a built in kernel flaw where network packets are forced to travel though multiple queues before they arrive at queue where they can be processed by the application. This imposes significant costs on multi-core systems due to queue locking costs. This flaw inherently diminishes the performance of Apache on multi-core system due to multiple threads spread across cores being forced to deal with these mutex (mutual exclusion) costs. For the sake of this experiment Apache had a separate instance on every core listening on different ports which is not a practical real world application but merely an attempt to implement better parallel execution on a traditional kernel. The patched kernel implementation of the network stack is also specific to the problem at hand, which is processing multiple short lived connections across multiple cores. Although this provides a performance increase in the given scenario, in more general applications network performance might suffer. These tests were also rigged to avoid bottlenecks in place by network and file storage hardware. Meaning, making the proposed modifications to the kernel wont necessarily produce the same increase in productivity as described in the article. This is very much evident in the test where performance degrades past 36 cores due to limitation of the networking hardware.

''Which is not a problem as the paper specifically states that they are testing what they can improve in spite of hardware limitation.'' - [[Rannath]]

====gmake: ''Section 5.6''====
Since the inherent nature of gmake makes it quite parallel, the testing and updating that were attempted on gmake resulted in essentially the same scalability results for both the stock and modified kernel. The only change that was found was that gmake spent slightly less time at the system level because of the changes that were made to the system's caching. As stated in the paper, the execution time of gmake relies quite heavily on the compiler that is uses with gmake, so depending on which compiler was chosen, gmake could run worse or even slightly better. In any case, there seems to be no fairness concerns when it comes to the scalability testing of gmake as the same application load-out was used for all of the tests.

====Conclusion: ''Sections 6 & 7''====
Given that all tests are more or less fair for the purposes of the benchmarks. They would support the Hypothesis that Linux can be made to scale, at least to 48 cores. Thus the conclusion is fair iff the rest of the paper is fair.

Now you just have to fill in how fair the rest of the paper is.

===Style===
Style Criterion (feel free to add I have no idea what should go here):
- does the paper present information out of order?
- does the paper present needless information?
- does the paper have any sections that are inherently confusing? Wrong?
- is the paper easy to read through, or does it change subjects repeatedly?
- does the paper have too many "long-winded" sentences, making it seem like the authors are just trying to add extra words to make it seem more important? - I think maybe limit this to run-on sentences.
- Check for grammar

Everything seems to be in logical order. I couldn't find any needless info. Nothing inherently confusing or wrong. Nothing bad on the grammar front either. - Rannath

Some acronyms aren't explained before they are used, so some people reading the paper may get confused as to what they mean (e.g. Linux TLB). Since this paper is meant to be formal, acronyms should be explained, with some exceptions like OS and IBM. - Daniel B.

Your example has no impact on the paper, it was in the "look here for more info" section. Most people wouldn't know what a "translation look-aside buffer" is either.

==References==

[3] memcached's wiki: http://code.google.com/p/memcached/wiki/FAQ#Can_I_use_different_size_caches_across_servers_and_will_memcache

You will almost certainly have to refer to other resources; please cite these resources in the style of citation of the papers assigned (inlined numbered references). Place your bibliographic entries in this section.

'''the paper itself doesn't need to be referenced more than once as this is a critique of the paper...'''
[1] Silas Boyd-Wickizer et al. "An Analysis of Linux Scalability to Many Cores". In ''OSDI '10, 9th USENIX Symposium on OS Design and Implementation'', Vancouver, BC, Canada, 2010. http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf.

gmake:

[http://www.gnu.org/software/make/manual/make.html gmake Manual]

[http://www.gnu.org/software/make/ gmake Main Page]

==Deprecated==
===Background Concepts===
* Exim: ''Section 3.1'':
**Exim is a mail server for Unix. It's fairly parallel. The server forks a new process for each connection and twice to deliver each message. It spends 69% of its time in the kernel on a single core.

* PostgreSQL: ''Section 3.4'':
**As implied by the name PostgreSQL is a SQL database. PostgreSQL starts a separate process for each connection and uses kernel locking interfaces extensively to provide concurrent access to the database. Due to bottlenecks introduced in its code and in the kernel code, the amount of time PostgreSQL spends in the kernel increases very rapidly with addition of new cores. On a single core system PostgreSQL spends only 1.5% of its time in the kernel. On a 48 core system the execution time in the kernel jumps to 82%.

Talk:COMP 3000 Essay 2 2010 Question 1

2010-12-02T17:44:19Z

Kkashigi: /* Multicore packet processing: Section 4.2 */

=Class and Notices=
(Nov. 30, 2010) Prof. Anil stated that we should focus on the 3 easiest to understand parts in section 5 and elaborate on them.

- Also, I, Daniel B., work Thursday night, so I will be finishing up as much of my part as I can for the essay before Thursday morning's class, then maybe we can all meet up in a lab in HP and put the finishing touches on the essay. I will be available online Wednesday night from about 6:30pm onwards and will be in the game dev lab or CCSS lounge Wednesday morning from about 11am to 2pm if anyone would like to meet up with me at those times.

- [[I suggest we meet up Thursday morning after Operating Systems in order to discuss and finalize the essay. Maybe we can even designate a lab for the group to meet up in. Any suggestions?]] - Daniel B.

- HP 3115 since there wont be a class in there (as its our tutorial and we know there won't be anyone there)
-- Go to Wireless Lab next to CCSS Lounge. Andrew and Dan B. will be there.

- If its all the same to you guys mind if I just join you via msn or iirc? Or phone if you really want. -Rannath

- I'm working today, but I'll be at a computer reading this page/contributing to my section. Depending on how busy I am, I should be able to get some significant writing in before 4pm today on my section and any additional sections required. RP

- I wont be there either. that does not mean i wont/cant contribute. I'll be on msn or you can just email me. -kirill

=Group members=
Patrick Young [mailto:Rannath@gmail.com Rannath@gmail.com]

Daniel Beimers [mailto:demongyro@gmail.com demongyro@gmail.com]

Andrew Bown [mailto:abown2@connect.carleton.ca abown2@connect.carleton.ca]

Kirill Kashigin [mailto:k.kashigin@gmail.com k.kashigin@gmail.com]

Rovic Perdon [mailto:rperdon@gmail.com rperdon@gmail.com]

Daniel Sont [mailto:dan.sont@gmail.com dan.sont@gmail.com]

=Methodology=
We should probably have our work verified by at least one group member before posting to the actual page

=To Do=
*Improve the grammar/structure of the paper section & add links to supplementary info
*flesh out the whole lot

===Claim Sections===
* So here is the claims and unclaimed section. Add your name next to one if you want to take it on.
** gmake - Daniel B.
** memcached - Rannath
** Apache - Kirill
** [[(Exim, PostgreSQL, Metis, and Psearchy will not be needed as the professor said we only need to explain 3)]]
** Research Problem - Andrew
** Contribution - Rovic
** Essay Conclusion (also discussion) - Everyone
** Critic, Style - Everyone
** References - Everyone

=Essay=
==Paper - DONE!!!==
This paper was authored by - Silas Boyd-Wickizer, Austin T. Clements, Yandong Mao, Aleksey Pesterev, M. Frans Kaashoek, Robert Morris, and Nickolai Zeldovich.

They all work at MIT CSAIL.

The paper: [http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf An Analysis of Linux Scalability to Many Cores]

==Background Concepts - DONE!!!==
===memcached: ''Section 3.2''===
memcached is an in-memory hash table server. One instance of memcached running on many different cores is bottlenecked by an internal lock, which is avoided by the MIT team by running one instance per core. Clients each connect to a single instance of memcached, allowing the server to simulate parallelism without needing to make major changes to the application or kernel. With few requests, memcached spends 80% of its time in the kernel on one core, mostly processing packets.[1]

===Apache: ''Section 3.3''===
Apache is a web server that has been used in previous Linux scalability studies. In the case of this study, Apache has been configured to run a separate process on each core. Each process, in turn, has multiple threads (making it a perfect example of parallel programming). Each process uses one of their threads to accepting incoming connections and others are used to process these connections. On a single core processor, Apache spends 60% of its execution time in the kernel.[1]

===gmake: ''Section 3.5''===
gmake is an unofficial default benchmark in the Linux community which is used in this paper to build the Linux kernel. gmake takes a file called a makefile and processes its recipes for the requisite files to determine how and when to remake or recompile code. With a simple command -j or --jobs, gmake can process many of these recipes in parallel. Since gmake creates more processes than cores, it can make proper use of multiple cores to process the recipes.[2] Since gmake involves much reading and writing, in order to prevent bottlenecks due to the filesystem or hardware, the test cases use an in-memory filesystem tmpfs, which gives them a backdoor around the bottlenecks for testing purposes. In addition to this, gmake is limited in scalability due to the serial processes that run at the beginning and end of its execution, which limits its scalability to a small degree. gmake spends much of its execution time with its compiler, processing the recipes and recompiling code, but still spend 7.6% of its time in system time.[1]

[2] http://www.gnu.org/software/make/manual/make.html

==Research problem==
my references are just below because it is easier for numbering the data later.

As technological progress the number of core a main processor can have is increasing at an impressive rate. Soon personal computers will have so many cores that scalability will be an issue. There has to be a way that standard user level Linux kernel will scale with a 48-core system[1]. The problem with a standard Linux operating is they are not designed for massive scalability which will soon be a problem. The issue with scalability is that a solo core will perform much more work compared to a single core working with 47 other cores. Although traditional logic that situation makes sense because 48 cores are dividing the work. But when processing information a process the main goal is to finish so as long as possible every core should be doing a much work as possible.

To fix those scalability issues it is necessary to focus on three major areas: the Linux kernel, user level design and how application use of kernel services. The Linux kernel can be improved be to improve sharing and have the advantage of recent iterations are beginning to implement scalability features. On the user level design applications can be improved so that there is more focus on parallelism since some programs have not implements those improved features. The final aspect of improving scalability is how an application uses kernel services to share resources better so that different aspects of the program are not conflicting over the same services. All of the bottlenecks are found actually only take a little work to avoid.[1]

This research is based on much research which was created before in the development of scalability for UNIX system. The major developments from shared memory machines [2], wait-free synchronization to fast message passing have created a base set of techniques which can be used to improve scalability. These techniques have been incorporated in all major operation system including Linux, Mac OS X and Windows. Linux has been improved with kernel subsystems such as Read-Copy-Update which an algorithm for which is used to avoid locks and atomic instructions which lower scalability.[3] The is also an excellent base a research on Linux scalability studies to base this research paper. These paper include a on doing scalability on a 32-core machine. [4] That research can improve the results by learning from the experiments already performed by researchers. This research also aid identifying bottlenecks which speed up researching solutions for those bottlenecks.

[2] J. Kuskin, D. Ofelt, M. Heinrich, J. Heinlein, R. Simoni, K. Gharachorloo, J. Chapin, D. Nakahira, J. Baxter, M. Horowitz, A. Gupta, M. Rosenblum, and J. Hennessy. The Stanford FLASH multiprocessor. In Proc. of the 21st ISCA, pages 302–313,1994.

[3] P. E. McKenney, D. Sarma, A. Arcangeli, A. Kleen, O. Krieger, and R. Russell. Read-copy-update. In Proceedings of the Linux Symposium 2002, pages 338-367, Ottawa Ontario, June 2002

[4] C. Yan, Y. Chen, and S. Yuanchun. OSMark: A benchmark suite for understanding parallel scalability of operating systems on large scale multi-cores. In 2009 2nd International Conference on Computer Science and Information Technology, pages 313–317, 2009

==Contribution==
What was implemented? Why is it any better than what came before?

Summarize info from Section 4.2 onwards, maybe put graphs from Section 5 here to provide support for improvements (if that isn't unethical/illegal)?

- So long as we cite the paper and don't pretend the graphs are ours, we are ok, since we are writing an explanation/critic of the paper.

All contributions in this paper are the result of the identification and removal or marginalization of bottlenecks.

===What hinders scalability: ''Section 4.1''===
*The percentage of serialization in a program has a lot to do with how much an application can be sped up. This is Amdahl's Law
** Amdahl's Law states that a parallel program can only be sped up by the inverse of the proportion of the program that cannot be made parallel (e.g. 25%(.25) non-parallel --> limit of 4x speedup) (I can't get this to sound right someone fix it please -[[Rannath]] <- I will fix [[Daniel B.]]
*Types of serializing interactions found in the MOSBENCH apps:
**Locking of shared data structure as the number of cores increase leads to an increase in lock wait time
**Writing to shared memory as the number of cores increase leads to an increase in the execution time of the cache coherence protocol
**Competing for space in shared hardware cache as the number of cores increase leads to an increase in cache miss rate
**Competing for shared hardware resources as the number of cores increase leads to time lost waiting for resources
**Not enough tasks for cores leads to idle cores

===Multicore packet processing: ''Section 4.2''===
Linux packet processing technique requires the packets to travel along several queues before it finally becomes available for the application to use. This technique works well for most general socket applications. In recent kernel releases Linux takes advantage of multiple hardware queues (when available on the given network interface) or Receive Packet Steering[1] to direct packet flow onto different cores for processing. Or even go as far as directing packet flow to the core on which the application is running using Receive Flow Steering[2] for even better performance. Linux also attempts to increase performance using a sampling technique where it checks every 20th outgoing packet and directs flow based on its hash. This poses a problem for short lived connections like those associated with Apache since there is great potential for packets to be misdirected.

In general this technique performs poorly when there are numerous open connections spread across multiple cores due to mutex (mutual exclusion) delays and cache misses. In such scenarios its better to process all connections, with associated packets and queues, on one core to avoid said issues. The patched kernel's implementation proposed in this article uses multiple hardware queues (which can be accomplished through Receive Packet Sharing) to direct all packets from a given connection to the same core. In turn Apache is modified to only accept connections if the thread dedicated to processing them is on the same core. If the current core's queue is found to be empty it will attempt to obtain work from queues located on different cores. This configuration is ideal for numerous short connections as all the work for them in accomplished quickly on one core avoiding unnecessary mutex delays associated with packet queues and inter-core cache misses.

[1] J. Corbet. Receive Packet Steering, November 2009. http://lwn.net/Articles/362339/.

[2] J. Edge. Receive Flow Steering, April 2010. http://lwn.net/Articles/382428/.

===Sloppy counters: ''Section 4.3''===
Bottlenecks were encountered when the applications undergoing testing were referencing and updating shared counters for multiple cores. The solution in the paper is to use sloppy counters, which gets each core to track its own separate counts of references and uses a central shared counter to keep all counts on track. This is ideal because each core updates its counts by modifying its per-core counter, usually only needing access to its own local cache, cutting down on waiting for locks or serialization. Sloppy counters are also backwards-compatible with existing shared-counter code, making its implementation much easier to accomplish. The main disadvantages of the sloppy counters are that in situations where object de-allocation occurs often, because the de-allocation itself is an expensive operation, and the counters use up space proportional to the number of cores.

===Lock-free comparison: ''Section 4.4''===
This section describes a specific instance of unnecessary locking.

===Per-Core Data Structures: ''Section 4.5''===
Three centralized data structures were causing bottlenecks - a per-superblock list of open files, vfsmount table, the packet buffers free list. Each data structure was decentralized to per-core versions of itself. In the case of vfsmount the central data structure was maintained, and any per-core misses got written from the central table to the per-core table.

===Eliminating false sharing: ''Section 4.6''===
Misplaced variables on the cache cause different cores to request the same line to be read and written at the same time often enough to significantly impact performance. By moving the often written variable to another line the bottleneck was removed.

===Avoiding unnecessary locking: ''Section 4.7''===
Many locks/mutexes have special cases where they don't need to lock. Likewise mutexes can be split from locking the whole data structure to locking a part of it. Both these changes remove or reduce bottlenecks.

==Conclusion: ''Sections 6 & 7''==
===Work in Progress===

====[[Rovic P.]]====
-I'm just using this as a notepad, do not copy/paste this section, I will put in a properly written set of paragraphs which will fit with the contribution questions asked. -RP

This research contributes by evaluating the scalability discrepancies of applications programming and kernel programming. Key discoveries in this research show the effectiveness of the kernel in handling scaling amongst CPU cores. This has also shown that scaling in application programming should be more the focus. It has been shown that simple scaling techniques (list techniques) such as programming parallelism (look up more stuff to back this up and quotes). (Sloppy counter effectiveness, possible positive contributions, what has been used (internet search), what hasn’t been used.) Read conclusion, 2nd paragraph.

One reason the
required changes are modest is that stock Linux already
incorporates many modifications to improve scalability.
More speculatively, perhaps it is the case that Linux’s
system-call API is well suited to an implementation that
avoids unnecessary contention over kernel objects.

====[[Rannath]]====
Everything so far indicates that the MOSBENCH applications can scale to 48 cores. This scaling required a few modest changes to remove bottlenecks. The MIT team speculate that that trend will continue as the number of cores increase. They also state that things not bottlenecked by the CPU are harder to fix.

We can eliminate most kernel bottlenecks that the applications hits most often with minor changes. Most changes were well known methodology, with the exception of Sloppy counters. This study is limited by the removal of the IO bottleneck, but it does suggest that traditional implementations can be made scalable.

==Critique==
What is good and not-so-good about this paper? You may discuss both the style and content;
be sure to ground your discussion with specific references. Simple assertions that something is good or bad is not enough - you must explain why.
Since this is a "my implementation is better then your implementation" paper the "goodness" of content can be impartially determined by its fairness and the honesty of the authors.

===Content(Fairness): ''Section 5''===

====memcached: ''Section 5.3''====
memcached is treated with near perfect fairness in the paper. Its an in-memory service, so the ignored storage IO bottleneck does not affect it at all. Likewise the "stock" and "PK" implementations are given the same test suite, so there is no advantage given to either. memcached itself is non-scalable, so the MIT team was forced to run one instance per-core to keep up throughput. The FAQ at memcached.org's wiki suggests using multiple implementations per-server as a work around to another problem, which implies that running multiple instances of the server is the same, or nearly the same, as running one larger server [3]. In the end memcached was bottlenecked by the network card.

====Apache: ''Section 5.4''====
Linux has a built in kernel flaw where network packets are forced to travel though multiple queues before they arrive at queue where they can be processed by the application. This imposes significant costs on multi-core systems due to queue locking costs. This flaw inherently diminishes the performance of Apache on multi-core system due to multiple threads spread across cores being forced to deal with these mutex (mutual exclusion) costs. For the sake of this experiment Apache had a separate instance on every core listening on different ports which is not a practical real world application but merely an attempt to implement better parallel execution on a traditional kernel. The patched kernel implementation of the network stack is also specific to the problem at hand, which is processing multiple short lived connections across multiple cores. Although this provides a performance increase in the given scenario, in more general applications network performance might suffer. These tests were also rigged to avoid bottlenecks in place by network and file storage hardware. Meaning, making the proposed modifications to the kernel wont necessarily produce the same increase in productivity as described in the article. This is very much evident in the test where performance degrades past 36 cores due to limitation of the networking hardware.

''Which is not a problem as the paper specifically states that they are testing what they can improve in spite of hardware limitation.'' - [[Rannath]]

====gmake: ''Section 5.6''====
Since the inherent nature of gmake makes it quite parallel, the testing and updating that were attempted on gmake resulted in essentially the same scalability results for both the stock and modified kernel. The only change that was found was that gmake spent slightly less time at the system level because of the changes that were made to the system's caching. As stated in the paper, the execution time of gmake relies quite heavily on the compiler that is uses with gmake, so depending on which compiler was chosen, gmake could run worse or even slightly better. In any case, there seems to be no fairness concerns when it comes to the scalability testing of gmake as the same application load-out was used for all of the tests.

====Conclusion: ''Sections 6 & 7''====
Given that all tests are more or less fair for the purposes of the benchmarks. They would support the Hypothesis that Linux can be made to scale, at least to 48 cores. Thus the conclusion is fair iff the rest of the paper is fair.

Now you just have to fill in how fair the rest of the paper is.

===Style===
Style Criterion (feel free to add I have no idea what should go here):
- does the paper present information out of order?
- does the paper present needless information?
- does the paper have any sections that are inherently confusing? Wrong?
- is the paper easy to read through, or does it change subjects repeatedly?
- does the paper have too many "long-winded" sentences, making it seem like the authors are just trying to add extra words to make it seem more important? - I think maybe limit this to run-on sentences.
- Check for grammar

Everything seems to be in logical order. I couldn't find any needless info. Nothing inherently confusing or wrong. Nothing bad on the grammar front either. - Rannath

Some acronyms aren't explained before they are used, so some people reading the paper may get confused as to what they mean (e.g. Linux TLB). Since this paper is meant to be formal, acronyms should be explained, with some exceptions like OS and IBM. - Daniel B.

Your example has no impact on the paper, it was in the "look here for more info" section. Most people wouldn't know what a "translation look-aside buffer" is either.

==References==

[3] memcached's wiki: http://code.google.com/p/memcached/wiki/FAQ#Can_I_use_different_size_caches_across_servers_and_will_memcache

You will almost certainly have to refer to other resources; please cite these resources in the style of citation of the papers assigned (inlined numbered references). Place your bibliographic entries in this section.

'''the paper itself doesn't need to be referenced more than once as this is a critique of the paper...'''
[1] Silas Boyd-Wickizer et al. "An Analysis of Linux Scalability to Many Cores". In ''OSDI '10, 9th USENIX Symposium on OS Design and Implementation'', Vancouver, BC, Canada, 2010. http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf.

gmake:

[http://www.gnu.org/software/make/manual/make.html gmake Manual]

[http://www.gnu.org/software/make/ gmake Main Page]

==Deprecated==
===Background Concepts===
* Exim: ''Section 3.1'':
**Exim is a mail server for Unix. It's fairly parallel. The server forks a new process for each connection and twice to deliver each message. It spends 69% of its time in the kernel on a single core.

* PostgreSQL: ''Section 3.4'':
**As implied by the name PostgreSQL is a SQL database. PostgreSQL starts a separate process for each connection and uses kernel locking interfaces extensively to provide concurrent access to the database. Due to bottlenecks introduced in its code and in the kernel code, the amount of time PostgreSQL spends in the kernel increases very rapidly with addition of new cores. On a single core system PostgreSQL spends only 1.5% of its time in the kernel. On a 48 core system the execution time in the kernel jumps to 82%.

Talk:COMP 3000 Essay 2 2010 Question 1

2010-12-02T17:43:55Z

Kkashigi: /* Multicore packet processing: Section 4.2 */

=Class and Notices=
(Nov. 30, 2010) Prof. Anil stated that we should focus on the 3 easiest to understand parts in section 5 and elaborate on them.

- Also, I, Daniel B., work Thursday night, so I will be finishing up as much of my part as I can for the essay before Thursday morning's class, then maybe we can all meet up in a lab in HP and put the finishing touches on the essay. I will be available online Wednesday night from about 6:30pm onwards and will be in the game dev lab or CCSS lounge Wednesday morning from about 11am to 2pm if anyone would like to meet up with me at those times.

- [[I suggest we meet up Thursday morning after Operating Systems in order to discuss and finalize the essay. Maybe we can even designate a lab for the group to meet up in. Any suggestions?]] - Daniel B.

- HP 3115 since there wont be a class in there (as its our tutorial and we know there won't be anyone there)
-- Go to Wireless Lab next to CCSS Lounge. Andrew and Dan B. will be there.

- If its all the same to you guys mind if I just join you via msn or iirc? Or phone if you really want. -Rannath

- I'm working today, but I'll be at a computer reading this page/contributing to my section. Depending on how busy I am, I should be able to get some significant writing in before 4pm today on my section and any additional sections required. RP

- I wont be there either. that does not mean i wont/cant contribute. I'll be on msn or you can just email me. -kirill

=Group members=
Patrick Young [mailto:Rannath@gmail.com Rannath@gmail.com]

Daniel Beimers [mailto:demongyro@gmail.com demongyro@gmail.com]

Andrew Bown [mailto:abown2@connect.carleton.ca abown2@connect.carleton.ca]

Kirill Kashigin [mailto:k.kashigin@gmail.com k.kashigin@gmail.com]

Rovic Perdon [mailto:rperdon@gmail.com rperdon@gmail.com]

Daniel Sont [mailto:dan.sont@gmail.com dan.sont@gmail.com]

=Methodology=
We should probably have our work verified by at least one group member before posting to the actual page

=To Do=
*Improve the grammar/structure of the paper section & add links to supplementary info
*flesh out the whole lot

===Claim Sections===
* So here is the claims and unclaimed section. Add your name next to one if you want to take it on.
** gmake - Daniel B.
** memcached - Rannath
** Apache - Kirill
** [[(Exim, PostgreSQL, Metis, and Psearchy will not be needed as the professor said we only need to explain 3)]]
** Research Problem - Andrew
** Contribution - Rovic
** Essay Conclusion (also discussion) - Everyone
** Critic, Style - Everyone
** References - Everyone

=Essay=
==Paper - DONE!!!==
This paper was authored by - Silas Boyd-Wickizer, Austin T. Clements, Yandong Mao, Aleksey Pesterev, M. Frans Kaashoek, Robert Morris, and Nickolai Zeldovich.

They all work at MIT CSAIL.

The paper: [http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf An Analysis of Linux Scalability to Many Cores]

==Background Concepts - DONE!!!==
===memcached: ''Section 3.2''===
memcached is an in-memory hash table server. One instance of memcached running on many different cores is bottlenecked by an internal lock, which is avoided by the MIT team by running one instance per core. Clients each connect to a single instance of memcached, allowing the server to simulate parallelism without needing to make major changes to the application or kernel. With few requests, memcached spends 80% of its time in the kernel on one core, mostly processing packets.[1]

===Apache: ''Section 3.3''===
Apache is a web server that has been used in previous Linux scalability studies. In the case of this study, Apache has been configured to run a separate process on each core. Each process, in turn, has multiple threads (making it a perfect example of parallel programming). Each process uses one of their threads to accepting incoming connections and others are used to process these connections. On a single core processor, Apache spends 60% of its execution time in the kernel.[1]

===gmake: ''Section 3.5''===
gmake is an unofficial default benchmark in the Linux community which is used in this paper to build the Linux kernel. gmake takes a file called a makefile and processes its recipes for the requisite files to determine how and when to remake or recompile code. With a simple command -j or --jobs, gmake can process many of these recipes in parallel. Since gmake creates more processes than cores, it can make proper use of multiple cores to process the recipes.[2] Since gmake involves much reading and writing, in order to prevent bottlenecks due to the filesystem or hardware, the test cases use an in-memory filesystem tmpfs, which gives them a backdoor around the bottlenecks for testing purposes. In addition to this, gmake is limited in scalability due to the serial processes that run at the beginning and end of its execution, which limits its scalability to a small degree. gmake spends much of its execution time with its compiler, processing the recipes and recompiling code, but still spend 7.6% of its time in system time.[1]

[2] http://www.gnu.org/software/make/manual/make.html

==Research problem==
my references are just below because it is easier for numbering the data later.

As technological progress the number of core a main processor can have is increasing at an impressive rate. Soon personal computers will have so many cores that scalability will be an issue. There has to be a way that standard user level Linux kernel will scale with a 48-core system[1]. The problem with a standard Linux operating is they are not designed for massive scalability which will soon be a problem. The issue with scalability is that a solo core will perform much more work compared to a single core working with 47 other cores. Although traditional logic that situation makes sense because 48 cores are dividing the work. But when processing information a process the main goal is to finish so as long as possible every core should be doing a much work as possible.

To fix those scalability issues it is necessary to focus on three major areas: the Linux kernel, user level design and how application use of kernel services. The Linux kernel can be improved be to improve sharing and have the advantage of recent iterations are beginning to implement scalability features. On the user level design applications can be improved so that there is more focus on parallelism since some programs have not implements those improved features. The final aspect of improving scalability is how an application uses kernel services to share resources better so that different aspects of the program are not conflicting over the same services. All of the bottlenecks are found actually only take a little work to avoid.[1]

This research is based on much research which was created before in the development of scalability for UNIX system. The major developments from shared memory machines [2], wait-free synchronization to fast message passing have created a base set of techniques which can be used to improve scalability. These techniques have been incorporated in all major operation system including Linux, Mac OS X and Windows. Linux has been improved with kernel subsystems such as Read-Copy-Update which an algorithm for which is used to avoid locks and atomic instructions which lower scalability.[3] The is also an excellent base a research on Linux scalability studies to base this research paper. These paper include a on doing scalability on a 32-core machine. [4] That research can improve the results by learning from the experiments already performed by researchers. This research also aid identifying bottlenecks which speed up researching solutions for those bottlenecks.

[2] J. Kuskin, D. Ofelt, M. Heinrich, J. Heinlein, R. Simoni, K. Gharachorloo, J. Chapin, D. Nakahira, J. Baxter, M. Horowitz, A. Gupta, M. Rosenblum, and J. Hennessy. The Stanford FLASH multiprocessor. In Proc. of the 21st ISCA, pages 302–313,1994.

[3] P. E. McKenney, D. Sarma, A. Arcangeli, A. Kleen, O. Krieger, and R. Russell. Read-copy-update. In Proceedings of the Linux Symposium 2002, pages 338-367, Ottawa Ontario, June 2002

[4] C. Yan, Y. Chen, and S. Yuanchun. OSMark: A benchmark suite for understanding parallel scalability of operating systems on large scale multi-cores. In 2009 2nd International Conference on Computer Science and Information Technology, pages 313–317, 2009

==Contribution==
What was implemented? Why is it any better than what came before?

Summarize info from Section 4.2 onwards, maybe put graphs from Section 5 here to provide support for improvements (if that isn't unethical/illegal)?

- So long as we cite the paper and don't pretend the graphs are ours, we are ok, since we are writing an explanation/critic of the paper.

All contributions in this paper are the result of the identification and removal or marginalization of bottlenecks.

===What hinders scalability: ''Section 4.1''===
*The percentage of serialization in a program has a lot to do with how much an application can be sped up. This is Amdahl's Law
** Amdahl's Law states that a parallel program can only be sped up by the inverse of the proportion of the program that cannot be made parallel (e.g. 25%(.25) non-parallel --> limit of 4x speedup) (I can't get this to sound right someone fix it please -[[Rannath]] <- I will fix [[Daniel B.]]
*Types of serializing interactions found in the MOSBENCH apps:
**Locking of shared data structure as the number of cores increase leads to an increase in lock wait time
**Writing to shared memory as the number of cores increase leads to an increase in the execution time of the cache coherence protocol
**Competing for space in shared hardware cache as the number of cores increase leads to an increase in cache miss rate
**Competing for shared hardware resources as the number of cores increase leads to time lost waiting for resources
**Not enough tasks for cores leads to idle cores

===Multicore packet processing: ''Section 4.2''===
Linux packet processing technique requires the packets to travel along several queues before it finally becomes available for the application to use. This technique works well for most general socket applications. In recent kernel releases Linux takes advantage of multiple hardware queues (when available on the given network interface) or Receive Packet Steering[1] to direct packet flow onto different cores for processing. Or even go as far as directing packet flow to the core on which the application is running using Receive Flow Steering[2] for even better performance. Linux also attempts to increase performance using a sampling technique where it checks every 20th outgoing packet and directs flow based on its hash. This poses a problem for short lived connections like those associated with Apache since there is great potential for packets to be misdirected.

In general this technique performs poorly when there are numerous open connections spread across multiple cores due to mutex (mutual exclusion) delays and cache misses. In such scenarios its better to process all connections, with associated packets and queues, on one core to avoid said issues. The patched kernel's implementation proposed in this article uses multiple hardware queues (which can be accomplished through Receive Packet Sharing) to direct all packets from a given connection to the same core. In turn Apache is modified to only accept connections if the thread dedicated to processing them is on the same core. If the current core's queue is found to be empty it will attempt to obtain work from queues located on different cores. This configuration is ideal for numerous short connections as all the work for them in accomplished quickly on one core avoiding unnecessary mutex delays associated with packet queues and inter-core cache misses.

<ref>[1] J. Corbet. Receive Packet Steering, November 2009. http://lwn.net/Articles/362339/.</ref>

<ref>[2] J. Edge. Receive Flow Steering, April 2010. http://lwn.net/Articles/382428/.</ref>

===Sloppy counters: ''Section 4.3''===
Bottlenecks were encountered when the applications undergoing testing were referencing and updating shared counters for multiple cores. The solution in the paper is to use sloppy counters, which gets each core to track its own separate counts of references and uses a central shared counter to keep all counts on track. This is ideal because each core updates its counts by modifying its per-core counter, usually only needing access to its own local cache, cutting down on waiting for locks or serialization. Sloppy counters are also backwards-compatible with existing shared-counter code, making its implementation much easier to accomplish. The main disadvantages of the sloppy counters are that in situations where object de-allocation occurs often, because the de-allocation itself is an expensive operation, and the counters use up space proportional to the number of cores.

===Lock-free comparison: ''Section 4.4''===
This section describes a specific instance of unnecessary locking.

===Per-Core Data Structures: ''Section 4.5''===
Three centralized data structures were causing bottlenecks - a per-superblock list of open files, vfsmount table, the packet buffers free list. Each data structure was decentralized to per-core versions of itself. In the case of vfsmount the central data structure was maintained, and any per-core misses got written from the central table to the per-core table.

===Eliminating false sharing: ''Section 4.6''===
Misplaced variables on the cache cause different cores to request the same line to be read and written at the same time often enough to significantly impact performance. By moving the often written variable to another line the bottleneck was removed.

===Avoiding unnecessary locking: ''Section 4.7''===
Many locks/mutexes have special cases where they don't need to lock. Likewise mutexes can be split from locking the whole data structure to locking a part of it. Both these changes remove or reduce bottlenecks.

==Conclusion: ''Sections 6 & 7''==
===Work in Progress===

====[[Rovic P.]]====
-I'm just using this as a notepad, do not copy/paste this section, I will put in a properly written set of paragraphs which will fit with the contribution questions asked. -RP

This research contributes by evaluating the scalability discrepancies of applications programming and kernel programming. Key discoveries in this research show the effectiveness of the kernel in handling scaling amongst CPU cores. This has also shown that scaling in application programming should be more the focus. It has been shown that simple scaling techniques (list techniques) such as programming parallelism (look up more stuff to back this up and quotes). (Sloppy counter effectiveness, possible positive contributions, what has been used (internet search), what hasn’t been used.) Read conclusion, 2nd paragraph.

One reason the
required changes are modest is that stock Linux already
incorporates many modifications to improve scalability.
More speculatively, perhaps it is the case that Linux’s
system-call API is well suited to an implementation that
avoids unnecessary contention over kernel objects.

====[[Rannath]]====
Everything so far indicates that the MOSBENCH applications can scale to 48 cores. This scaling required a few modest changes to remove bottlenecks. The MIT team speculate that that trend will continue as the number of cores increase. They also state that things not bottlenecked by the CPU are harder to fix.

We can eliminate most kernel bottlenecks that the applications hits most often with minor changes. Most changes were well known methodology, with the exception of Sloppy counters. This study is limited by the removal of the IO bottleneck, but it does suggest that traditional implementations can be made scalable.

==Critique==
What is good and not-so-good about this paper? You may discuss both the style and content;
be sure to ground your discussion with specific references. Simple assertions that something is good or bad is not enough - you must explain why.
Since this is a "my implementation is better then your implementation" paper the "goodness" of content can be impartially determined by its fairness and the honesty of the authors.

===Content(Fairness): ''Section 5''===

====memcached: ''Section 5.3''====
memcached is treated with near perfect fairness in the paper. Its an in-memory service, so the ignored storage IO bottleneck does not affect it at all. Likewise the "stock" and "PK" implementations are given the same test suite, so there is no advantage given to either. memcached itself is non-scalable, so the MIT team was forced to run one instance per-core to keep up throughput. The FAQ at memcached.org's wiki suggests using multiple implementations per-server as a work around to another problem, which implies that running multiple instances of the server is the same, or nearly the same, as running one larger server [3]. In the end memcached was bottlenecked by the network card.

====Apache: ''Section 5.4''====
Linux has a built in kernel flaw where network packets are forced to travel though multiple queues before they arrive at queue where they can be processed by the application. This imposes significant costs on multi-core systems due to queue locking costs. This flaw inherently diminishes the performance of Apache on multi-core system due to multiple threads spread across cores being forced to deal with these mutex (mutual exclusion) costs. For the sake of this experiment Apache had a separate instance on every core listening on different ports which is not a practical real world application but merely an attempt to implement better parallel execution on a traditional kernel. The patched kernel implementation of the network stack is also specific to the problem at hand, which is processing multiple short lived connections across multiple cores. Although this provides a performance increase in the given scenario, in more general applications network performance might suffer. These tests were also rigged to avoid bottlenecks in place by network and file storage hardware. Meaning, making the proposed modifications to the kernel wont necessarily produce the same increase in productivity as described in the article. This is very much evident in the test where performance degrades past 36 cores due to limitation of the networking hardware.

''Which is not a problem as the paper specifically states that they are testing what they can improve in spite of hardware limitation.'' - [[Rannath]]

====gmake: ''Section 5.6''====
Since the inherent nature of gmake makes it quite parallel, the testing and updating that were attempted on gmake resulted in essentially the same scalability results for both the stock and modified kernel. The only change that was found was that gmake spent slightly less time at the system level because of the changes that were made to the system's caching. As stated in the paper, the execution time of gmake relies quite heavily on the compiler that is uses with gmake, so depending on which compiler was chosen, gmake could run worse or even slightly better. In any case, there seems to be no fairness concerns when it comes to the scalability testing of gmake as the same application load-out was used for all of the tests.

====Conclusion: ''Sections 6 & 7''====
Given that all tests are more or less fair for the purposes of the benchmarks. They would support the Hypothesis that Linux can be made to scale, at least to 48 cores. Thus the conclusion is fair iff the rest of the paper is fair.

Now you just have to fill in how fair the rest of the paper is.

===Style===
Style Criterion (feel free to add I have no idea what should go here):
- does the paper present information out of order?
- does the paper present needless information?
- does the paper have any sections that are inherently confusing? Wrong?
- is the paper easy to read through, or does it change subjects repeatedly?
- does the paper have too many "long-winded" sentences, making it seem like the authors are just trying to add extra words to make it seem more important? - I think maybe limit this to run-on sentences.
- Check for grammar

Everything seems to be in logical order. I couldn't find any needless info. Nothing inherently confusing or wrong. Nothing bad on the grammar front either. - Rannath

Some acronyms aren't explained before they are used, so some people reading the paper may get confused as to what they mean (e.g. Linux TLB). Since this paper is meant to be formal, acronyms should be explained, with some exceptions like OS and IBM. - Daniel B.

Your example has no impact on the paper, it was in the "look here for more info" section. Most people wouldn't know what a "translation look-aside buffer" is either.

==References==

[3] memcached's wiki: http://code.google.com/p/memcached/wiki/FAQ#Can_I_use_different_size_caches_across_servers_and_will_memcache

You will almost certainly have to refer to other resources; please cite these resources in the style of citation of the papers assigned (inlined numbered references). Place your bibliographic entries in this section.

'''the paper itself doesn't need to be referenced more than once as this is a critique of the paper...'''
[1] Silas Boyd-Wickizer et al. "An Analysis of Linux Scalability to Many Cores". In ''OSDI '10, 9th USENIX Symposium on OS Design and Implementation'', Vancouver, BC, Canada, 2010. http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf.

gmake:

[http://www.gnu.org/software/make/manual/make.html gmake Manual]

[http://www.gnu.org/software/make/ gmake Main Page]

==Deprecated==
===Background Concepts===
* Exim: ''Section 3.1'':
**Exim is a mail server for Unix. It's fairly parallel. The server forks a new process for each connection and twice to deliver each message. It spends 69% of its time in the kernel on a single core.

* PostgreSQL: ''Section 3.4'':
**As implied by the name PostgreSQL is a SQL database. PostgreSQL starts a separate process for each connection and uses kernel locking interfaces extensively to provide concurrent access to the database. Due to bottlenecks introduced in its code and in the kernel code, the amount of time PostgreSQL spends in the kernel increases very rapidly with addition of new cores. On a single core system PostgreSQL spends only 1.5% of its time in the kernel. On a 48 core system the execution time in the kernel jumps to 82%.

Talk:COMP 3000 Essay 2 2010 Question 1

2010-12-02T17:39:18Z

Kkashigi: /* Multicore packet processing: Section 4.2 */

=Class and Notices=
(Nov. 30, 2010) Prof. Anil stated that we should focus on the 3 easiest to understand parts in section 5 and elaborate on them.

- Also, I, Daniel B., work Thursday night, so I will be finishing up as much of my part as I can for the essay before Thursday morning's class, then maybe we can all meet up in a lab in HP and put the finishing touches on the essay. I will be available online Wednesday night from about 6:30pm onwards and will be in the game dev lab or CCSS lounge Wednesday morning from about 11am to 2pm if anyone would like to meet up with me at those times.

- [[I suggest we meet up Thursday morning after Operating Systems in order to discuss and finalize the essay. Maybe we can even designate a lab for the group to meet up in. Any suggestions?]] - Daniel B.

- HP 3115 since there wont be a class in there (as its our tutorial and we know there won't be anyone there)
-- Go to Wireless Lab next to CCSS Lounge. Andrew and Dan B. will be there.

- If its all the same to you guys mind if I just join you via msn or iirc? Or phone if you really want. -Rannath

- I'm working today, but I'll be at a computer reading this page/contributing to my section. Depending on how busy I am, I should be able to get some significant writing in before 4pm today on my section and any additional sections required. RP

- I wont be there either. that does not mean i wont/cant contribute. I'll be on msn or you can just email me. -kirill

=Group members=
Patrick Young [mailto:Rannath@gmail.com Rannath@gmail.com]

Daniel Beimers [mailto:demongyro@gmail.com demongyro@gmail.com]

Andrew Bown [mailto:abown2@connect.carleton.ca abown2@connect.carleton.ca]

Kirill Kashigin [mailto:k.kashigin@gmail.com k.kashigin@gmail.com]

Rovic Perdon [mailto:rperdon@gmail.com rperdon@gmail.com]

Daniel Sont [mailto:dan.sont@gmail.com dan.sont@gmail.com]

=Methodology=
We should probably have our work verified by at least one group member before posting to the actual page

=To Do=
*Improve the grammar/structure of the paper section & add links to supplementary info
*flesh out the whole lot

===Claim Sections===
* So here is the claims and unclaimed section. Add your name next to one if you want to take it on.
** gmake - Daniel B.
** memcached - Rannath
** Apache - Kirill
** [[(Exim, PostgreSQL, Metis, and Psearchy will not be needed as the professor said we only need to explain 3)]]
** Research Problem - Andrew
** Contribution - Rovic
** Essay Conclusion (also discussion) - Everyone
** Critic, Style - Everyone
** References - Everyone

=Essay=
==Paper - DONE!!!==
This paper was authored by - Silas Boyd-Wickizer, Austin T. Clements, Yandong Mao, Aleksey Pesterev, M. Frans Kaashoek, Robert Morris, and Nickolai Zeldovich.

They all work at MIT CSAIL.

The paper: [http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf An Analysis of Linux Scalability to Many Cores]

==Background Concepts==
===memcached: ''Section 3.2''===
memcached is an in-memory hash table server. One instance of memcached running on many different cores is bottlenecked by an internal lock, which is avoided by the MIT team by running one instance per core. Clients each connect to a single instance of memcached, allowing the server to simulate parallelism without needing to make major changes to the application or kernel. With few requests, memcached spends 80% of its time in the kernel on one core, mostly processing packets.[1]

===Apache: ''Section 3.3''===
Apache is a web server that has been used in previous Linux scalability studies. In the case of this study, Apache has been configured to run a separate process on each core. Each process, in turn, has multiple threads (making it a perfect example of parallel programming). Each process uses one of their threads to accepting incoming connections and others are used to process these connections. On a single core processor, Apache spends 60% of its execution time in the kernel.[1]

===gmake: ''Section 3.5''===
gmake is an unofficial default benchmark in the Linux community which is used in this paper to build the Linux kernel. gmake takes a file called a makefile and processes its recipes for the requisite files to determine how and when to remake or recompile code. With a simple command -j or --jobs, gmake can process many of these recipes in parallel. Since gmake creates more processes than cores, it can make proper use of multiple cores to process the recipes.[2] Since gmake involves much reading and writing, in order to prevent bottlenecks due to the filesystem or hardware, the test cases use an in-memory filesystem tmpfs, which gives them a backdoor around the bottlenecks for testing purposes. In addition to this, gmake is limited in scalability due to the serial processes that run at the beginning and end of its execution, which limits its scalability to a small degree. gmake spends much of its execution time with its compiler, processing the recipes and recompiling code, but still spend 7.6% of its time in system time.[1]

[2] http://www.gnu.org/software/make/manual/make.html

==Research problem==
my references are just below because it is easier for numbering the data later.

As technological progress the number of core a main processor can have is increasing at an impressive rate. Soon personal computers will have so many cores that scalability will be an issue. There has to be a way that standard user level Linux kernel will scale with a 48-core system[1]. The problem with a standard Linux operating is they are not designed for massive scalability which will soon be a problem. The issue with scalability is that a solo core will perform much more work compared to a single core working with 47 other cores. Although traditional logic that situation makes sense because 48 cores are dividing the work. But when processing information a process the main goal is to finish so as long as possible every core should be doing a much work as possible.

To fix those scalability issues it is necessary to focus on three major areas: the Linux kernel, user level design and how application use of kernel services. The Linux kernel can be improved be to improve sharing and have the advantage of recent iterations are beginning to implement scalability features. On the user level design applications can be improved so that there is more focus on parallelism since some programs have not implements those improved features. The final aspect of improving scalability is how an application uses kernel services to share resources better so that different aspects of the program are not conflicting over the same services. All of the bottlenecks are found actually only take a little work to avoid.[1]

This research is based on much research which was created before in the development of scalability for UNIX system. The major developments from shared memory machines [2], wait-free synchronization to fast message passing have created a base set of techniques which can be used to improve scalability. These techniques have been incorporated in all major operation system including Linux, Mac OS X and Windows. Linux has been improved with kernel subsystems such as Read-Copy-Update which an algorithm for which is used to avoid locks and atomic instructions which lower scalability.[3] The is also an excellent base a research on Linux scalability studies to base this research paper. These paper include a on doing scalability on a 32-core machine. [4] That research can improve the results by learning from the experiments already performed by researchers. This research also aid identifying bottlenecks which speed up researching solutions for those bottlenecks.

[2] J. Kuskin, D. Ofelt, M. Heinrich, J. Heinlein, R. Simoni, K. Gharachorloo, J. Chapin, D. Nakahira, J. Baxter, M. Horowitz, A. Gupta, M. Rosenblum, and J. Hennessy. The Stanford FLASH multiprocessor. In Proc. of the 21st ISCA, pages 302–313,1994.

[3] P. E. McKenney, D. Sarma, A. Arcangeli, A. Kleen, O. Krieger, and R. Russell. Read-copy-update. In Proceedings of the Linux Symposium 2002, pages 338-367, Ottawa Ontario, June 2002

[4] C. Yan, Y. Chen, and S. Yuanchun. OSMark: A benchmark suite for understanding parallel scalability of operating systems on large scale multi-cores. In 2009 2nd International Conference on Computer Science and Information Technology, pages 313–317, 2009

==Contribution==
What was implemented? Why is it any better than what came before?

Summarize info from Section 4.2 onwards, maybe put graphs from Section 5 here to provide support for improvements (if that isn't unethical/illegal)?

- So long as we cite the paper and don't pretend the graphs are ours, we are ok, since we are writing an explanation/critic of the paper.

All contributions in this paper are the result of the identification and removal or marginalization of bottlenecks.

===What hinders scalability: ''Section 4.1''===
*The percentage of serialization in a program has a lot to do with how much an application can be sped up. This is Amdahl's Law
** Amdahl's Law states that a parallel program can only be sped up by the inverse of the proportion of the program that cannot be made parallel (e.g. 25%(.25) non-parallel --> limit of 4x speedup) (I can't get this to sound right someone fix it please -[[Rannath]] <- I will fix [[Daniel B.]]
*Types of serializing interactions found in the MOSBENCH apps:
**Locking of shared data structure as the number of cores increase leads to an increase in lock wait time
**Writing to shared memory as the number of cores increase leads to an increase in the execution time of the cache coherence protocol
**Competing for space in shared hardware cache as the number of cores increase leads to an increase in cache miss rate
**Competing for shared hardware resources as the number of cores increase leads to time lost waiting for resources
**Not enough tasks for cores leads to idle cores

===Multicore packet processing: ''Section 4.2''===
Linux packet processing technique requires the packets to travel along several queues before it finally becomes available for the application to use. This technique works well for most general socket applications. In recent kernel releases Linux takes advantage of multiple hardware queues (when available on the given network interface) or Receive Packet Steering[1] to direct packet flow onto different cores for processing. Or even go as far as directing packet flow to the core on which the application is running using Receive Flow Steering[2] for even better performance. Linux also attempts to increase performance using a sampling technique where it checks every 20th outgoing packet and directs flow based on its hash. This poses a problem for short lived connections like those associated with Apache since there is great potential for packets to be misdirected.

In general this technique performs poorly when there are numerous open connections spread across multiple cores due to mutex (mutual exclusion) delays and cache misses. In such scenarios its better to process all connections, with associated packets and queues, on one core to avoid said issues. The patched kernel's implementation proposed in this article uses multiple hardware queues (which can be accomplished through Receive Packet Sharing) to direct all packets from a given connection to the same core. In turn Apache is modified to only accept connections if the thread dedicated to processing them is on the same core. If the current core's queue is found to be empty it will attempt to obtain work from queues located on different cores. This configuration is ideal for numerous short connections as all the work for them in accomplished quickly on one core avoiding unnecessary mutex delays associated with packet queues and inter-core cache misses.

[1] J. Corbet. Receive Packet Steering, November 2009. http://lwn.net/Articles/362339/.

[2] J. Edge. Receive Flow Steering, April 2010. http://lwn.net/Articles/382428/.

===Sloppy counters: ''Section 4.3''===
Bottlenecks were encountered when the applications undergoing testing were referencing and updating shared counters for multiple cores. The solution in the paper is to use sloppy counters, which gets each core to track its own separate counts of references and uses a central shared counter to keep all counts on track. This is ideal because each core updates its counts by modifying its per-core counter, usually only needing access to its own local cache, cutting down on waiting for locks or serialization. Sloppy counters are also backwards-compatible with existing shared-counter code, making its implementation much easier to accomplish. The main disadvantages of the sloppy counters are that in situations where object de-allocation occurs often, because the de-allocation itself is an expensive operation, and the counters use up space proportional to the number of cores.

===Lock-free comparison: ''Section 4.4''===
This section describes a specific instance of unnecessary locking.

===Per-Core Data Structures: ''Section 4.5''===
Three centralized data structures were causing bottlenecks - a per-superblock list of open files, vfsmount table, the packet buffers free list. Each data structure was decentralized to per-core versions of itself. In the case of vfsmount the central data structure was maintained, and any per-core misses got written from the central table to the per-core table.

===Eliminating false sharing: ''Section 4.6''===
Misplaced variables on the cache cause different cores to request the same line to be read and written at the same time often enough to significantly impact performance. By moving the often written variable to another line the bottleneck was removed.

===Avoiding unnecessary locking: ''Section 4.7''===
Many locks/mutexes have special cases where they don't need to lock. Likewise mutexes can be split from locking the whole data structure to locking a part of it. Both these changes remove or reduce bottlenecks.

==Conclusion: ''Sections 6 & 7''==
===Work in Progress===

====[[Rovic P.]]====
-I'm just using this as a notepad, do not copy/paste this section, I will put in a properly written set of paragraphs which will fit with the contribution questions asked. -RP

This research contributes by evaluating the scalability discrepancies of applications programming and kernel programming. Key discoveries in this research show the effectiveness of the kernel in handling scaling amongst CPU cores. This has also shown that scaling in application programming should be more the focus. It has been shown that simple scaling techniques (list techniques) such as programming parallelism (look up more stuff to back this up and quotes). (Sloppy counter effectiveness, possible positive contributions, what has been used (internet search), what hasn’t been used.) Read conclusion, 2nd paragraph.

One reason the
required changes are modest is that stock Linux already
incorporates many modifications to improve scalability.
More speculatively, perhaps it is the case that Linux’s
system-call API is well suited to an implementation that
avoids unnecessary contention over kernel objects.

====[[Rannath]]====
Everything so far indicates that the MOSBENCH applications can scale to 48 cores. This scaling required a few modest changes to remove bottlenecks. The MIT team speculate that that trend will continue as the number of cores increase. They also state that things not bottlenecked by the CPU are harder to fix.

We can eliminate most kernel bottlenecks that the applications hits most often with minor changes. Most changes were well known methodology, with the exception of Sloppy counters. This study is limited by the removal of the IO bottleneck, but it does suggest that traditional implementations can be made scalable.

==Critique==
What is good and not-so-good about this paper? You may discuss both the style and content;
be sure to ground your discussion with specific references. Simple assertions that something is good or bad is not enough - you must explain why.
Since this is a "my implementation is better then your implementation" paper the "goodness" of content can be impartially determined by its fairness and the honesty of the authors.

===Content(Fairness): ''Section 5''===

====memcached: ''Section 5.3''====
memcached is treated with near perfect fairness in the paper. Its an in-memory service, so the ignored storage IO bottleneck does not affect it at all. Likewise the "stock" and "PK" implementations are given the same test suite, so there is no advantage given to either. memcached itself is non-scalable, so the MIT team was forced to run one instance per-core to keep up throughput. The FAQ at memcached.org's wiki suggests using multiple implementations per-server as a work around to another problem, which implies that running multiple instances of the server is the same, or nearly the same, as running one larger server [3]. In the end memcached was bottlenecked by the network card.

====Apache: ''Section 5.4''====
Linux has a built in kernel flaw where network packets are forced to travel though multiple queues before they arrive at queue where they can be processed by the application. This imposes significant costs on multi-core systems due to queue locking costs. This flaw inherently diminishes the performance of Apache on multi-core system due to multiple threads spread across cores being forced to deal with these mutex (mutual exclusion) costs. For the sake of this experiment Apache had a separate instance on every core listening on different ports which is not a practical real world application but merely an attempt to implement better parallel execution on a traditional kernel. The patched kernel implementation of the network stack is also specific to the problem at hand, which is processing multiple short lived connections across multiple cores. Although this provides a performance increase in the given scenario, in more general applications network performance might suffer. These tests were also rigged to avoid bottlenecks in place by network and file storage hardware. Meaning, making the proposed modifications to the kernel wont necessarily produce the same increase in productivity as described in the article. This is very much evident in the test where performance degrades past 36 cores due to limitation of the networking hardware.

''Which is not a problem as the paper specifically states that they are testing what they can improve in spite of hardware limitation.'' - [[Rannath]]

====gmake: ''Section 5.6''====
Since the inherent nature of gmake makes it quite parallel, the testing and updating that were attempted on gmake resulted in essentially the same scalability results for both the stock and modified kernel. The only change that was found was that gmake spent slightly less time at the system level because of the changes that were made to the system's caching. As stated in the paper, the execution time of gmake relies quite heavily on the compiler that is uses with gmake, so depending on which compiler was chosen, gmake could run worse or even slightly better. In any case, there seems to be no fairness concerns when it comes to the scalability testing of gmake as the same application load-out was used for all of the tests.

====Conclusion: ''Sections 6 & 7''====
Given that all tests are more or less fair for the purposes of the benchmarks. They would support the Hypothesis that Linux can be made to scale, at least to 48 cores. Thus the conclusion is fair iff the rest of the paper is fair.

Now you just have to fill in how fair the rest of the paper is.

===Style===
Style Criterion (feel free to add I have no idea what should go here):
- does the paper present information out of order?
- does the paper present needless information?
- does the paper have any sections that are inherently confusing? Wrong?
- is the paper easy to read through, or does it change subjects repeatedly?
- does the paper have too many "long-winded" sentences, making it seem like the authors are just trying to add extra words to make it seem more important? - I think maybe limit this to run-on sentences.
- Check for grammar

Everything seems to be in logical order. I couldn't find any needless info. Nothing inherently confusing or wrong. Nothing bad on the grammar front either. - Rannath

Some acronyms aren't explained before they are used, so some people reading the paper may get confused as to what they mean (e.g. Linux TLB). Since this paper is meant to be formal, acronyms should be explained, with some exceptions like OS and IBM. - Daniel B.

Your example has no impact on the paper, it was in the "look here for more info" section. Most people wouldn't know what a "translation look-aside buffer" is either.

==References==

[3] memcached's wiki: http://code.google.com/p/memcached/wiki/FAQ#Can_I_use_different_size_caches_across_servers_and_will_memcache

You will almost certainly have to refer to other resources; please cite these resources in the style of citation of the papers assigned (inlined numbered references). Place your bibliographic entries in this section.

'''the paper itself doesn't need to be referenced more than once as this is a critique of the paper...'''
[1] Silas Boyd-Wickizer et al. "An Analysis of Linux Scalability to Many Cores". In ''OSDI '10, 9th USENIX Symposium on OS Design and Implementation'', Vancouver, BC, Canada, 2010. http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf.

gmake:

[http://www.gnu.org/software/make/manual/make.html gmake Manual]

[http://www.gnu.org/software/make/ gmake Main Page]

==Deprecated==
===Background Concepts===
* Exim: ''Section 3.1'':
**Exim is a mail server for Unix. It's fairly parallel. The server forks a new process for each connection and twice to deliver each message. It spends 69% of its time in the kernel on a single core.

* PostgreSQL: ''Section 3.4'':
**As implied by the name PostgreSQL is a SQL database. PostgreSQL starts a separate process for each connection and uses kernel locking interfaces extensively to provide concurrent access to the database. Due to bottlenecks introduced in its code and in the kernel code, the amount of time PostgreSQL spends in the kernel increases very rapidly with addition of new cores. On a single core system PostgreSQL spends only 1.5% of its time in the kernel. On a 48 core system the execution time in the kernel jumps to 82%.

Talk:COMP 3000 Essay 2 2010 Question 1

2010-12-02T17:37:52Z

Kkashigi: /* Multicore packet processing: Section 4.2 */

=Class and Notices=
(Nov. 30, 2010) Prof. Anil stated that we should focus on the 3 easiest to understand parts in section 5 and elaborate on them.

- Also, I, Daniel B., work Thursday night, so I will be finishing up as much of my part as I can for the essay before Thursday morning's class, then maybe we can all meet up in a lab in HP and put the finishing touches on the essay. I will be available online Wednesday night from about 6:30pm onwards and will be in the game dev lab or CCSS lounge Wednesday morning from about 11am to 2pm if anyone would like to meet up with me at those times.

- [[I suggest we meet up Thursday morning after Operating Systems in order to discuss and finalize the essay. Maybe we can even designate a lab for the group to meet up in. Any suggestions?]] - Daniel B.

- HP 3115 since there wont be a class in there (as its our tutorial and we know there won't be anyone there)
-- Go to Wireless Lab next to CCSS Lounge. Andrew and Dan B. will be there.

- If its all the same to you guys mind if I just join you via msn or iirc? Or phone if you really want. -Rannath

- I'm working today, but I'll be at a computer reading this page/contributing to my section. Depending on how busy I am, I should be able to get some significant writing in before 4pm today on my section and any additional sections required. RP

- I wont be there either. that does not mean i wont/cant contribute. I'll be on msn or you can just email me. -kirill

=Group members=
Patrick Young [mailto:Rannath@gmail.com Rannath@gmail.com]

Daniel Beimers [mailto:demongyro@gmail.com demongyro@gmail.com]

Andrew Bown [mailto:abown2@connect.carleton.ca abown2@connect.carleton.ca]

Kirill Kashigin [mailto:k.kashigin@gmail.com k.kashigin@gmail.com]

Rovic Perdon [mailto:rperdon@gmail.com rperdon@gmail.com]

Daniel Sont [mailto:dan.sont@gmail.com dan.sont@gmail.com]

=Methodology=
We should probably have our work verified by at least one group member before posting to the actual page

=To Do=
*Improve the grammar/structure of the paper section & add links to supplementary info
*flesh out the whole lot

===Claim Sections===
* So here is the claims and unclaimed section. Add your name next to one if you want to take it on.
** gmake - Daniel B.
** memcached - Rannath
** Apache - Kirill
** [[(Exim, PostgreSQL, Metis, and Psearchy will not be needed as the professor said we only need to explain 3)]]
** Research Problem - Andrew
** Contribution - Rovic
** Essay Conclusion (also discussion) - Everyone
** Critic, Style - Everyone
** References - Everyone

=Essay=
==Paper - DONE!!!==
This paper was authored by - Silas Boyd-Wickizer, Austin T. Clements, Yandong Mao, Aleksey Pesterev, M. Frans Kaashoek, Robert Morris, and Nickolai Zeldovich.

They all work at MIT CSAIL.

The paper: [http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf An Analysis of Linux Scalability to Many Cores]

==Background Concepts==
===memcached: ''Section 3.2''===
memcached is an in-memory hash table server. One instance of memcached running on many different cores is bottlenecked by an internal lock, which is avoided by the MIT team by running one instance per core. Clients each connect to a single instance of memcached, allowing the server to simulate parallelism without needing to make major changes to the application or kernel. With few requests, memcached spends 80% of its time in the kernel on one core, mostly processing packets.[1]

===Apache: ''Section 3.3''===
Apache is a web server that has been used in previous Linux scalability studies. In the case of this study, Apache has been configured to run a separate process on each core. Each process, in turn, has multiple threads (making it a perfect example of parallel programming). Each process uses one of their threads to accepting incoming connections and others are used to process these connections. On a single core processor, Apache spends 60% of its execution time in the kernel.[1]

===gmake: ''Section 3.5''===
gmake is an unofficial default benchmark in the Linux community which is used in this paper to build the Linux kernel. gmake takes a file called a makefile and processes its recipes for the requisite files to determine how and when to remake or recompile code. With a simple command -j or --jobs, gmake can process many of these recipes in parallel. Since gmake creates more processes than cores, it can make proper use of multiple cores to process the recipes.[2] Since gmake involves much reading and writing, in order to prevent bottlenecks due to the filesystem or hardware, the test cases use an in-memory filesystem tmpfs, which gives them a backdoor around the bottlenecks for testing purposes. In addition to this, gmake is limited in scalability due to the serial processes that run at the beginning and end of its execution, which limits its scalability to a small degree. gmake spends much of its execution time with its compiler, processing the recipes and recompiling code, but still spend 7.6% of its time in system time.[1]

[2] http://www.gnu.org/software/make/manual/make.html

==Research problem==
my references are just below because it is easier for numbering the data later.

As technological progress the number of core a main processor can have is increasing at an impressive rate. Soon personal computers will have so many cores that scalability will be an issue. There has to be a way that standard user level Linux kernel will scale with a 48-core system[1]. The problem with a standard Linux operating is they are not designed for massive scalability which will soon be a problem. The issue with scalability is that a solo core will perform much more work compared to a single core working with 47 other cores. Although traditional logic that situation makes sense because 48 cores are dividing the work. But when processing information a process the main goal is to finish so as long as possible every core should be doing a much work as possible.

To fix those scalability issues it is necessary to focus on three major areas: the Linux kernel, user level design and how application use of kernel services. The Linux kernel can be improved be to improve sharing and have the advantage of recent iterations are beginning to implement scalability features. On the user level design applications can be improved so that there is more focus on parallelism since some programs have not implements those improved features. The final aspect of improving scalability is how an application uses kernel services to share resources better so that different aspects of the program are not conflicting over the same services. All of the bottlenecks are found actually only take a little work to avoid.[1]

This research is based on much research which was created before in the development of scalability for UNIX system. The major developments from shared memory machines [2], wait-free synchronization to fast message passing have created a base set of techniques which can be used to improve scalability. These techniques have been incorporated in all major operation system including Linux, Mac OS X and Windows. Linux has been improved with kernel subsystems such as Read-Copy-Update which an algorithm for which is used to avoid locks and atomic instructions which lower scalability.[3] The is also an excellent base a research on Linux scalability studies to base this research paper. These paper include a on doing scalability on a 32-core machine. [4] That research can improve the results by learning from the experiments already performed by researchers. This research also aid identifying bottlenecks which speed up researching solutions for those bottlenecks.

[2] J. Kuskin, D. Ofelt, M. Heinrich, J. Heinlein, R. Simoni, K. Gharachorloo, J. Chapin, D. Nakahira, J. Baxter, M. Horowitz, A. Gupta, M. Rosenblum, and J. Hennessy. The Stanford FLASH multiprocessor. In Proc. of the 21st ISCA, pages 302–313,1994.

[3] P. E. McKenney, D. Sarma, A. Arcangeli, A. Kleen, O. Krieger, and R. Russell. Read-copy-update. In Proceedings of the Linux Symposium 2002, pages 338-367, Ottawa Ontario, June 2002

[4] C. Yan, Y. Chen, and S. Yuanchun. OSMark: A benchmark suite for understanding parallel scalability of operating systems on large scale multi-cores. In 2009 2nd International Conference on Computer Science and Information Technology, pages 313–317, 2009

==Contribution==
What was implemented? Why is it any better than what came before?

Summarize info from Section 4.2 onwards, maybe put graphs from Section 5 here to provide support for improvements (if that isn't unethical/illegal)?

- So long as we cite the paper and don't pretend the graphs are ours, we are ok, since we are writing an explanation/critic of the paper.

All contributions in this paper are the result of the identification and removal or marginalization of bottlenecks.

===What hinders scalability: ''Section 4.1''===
*The percentage of serialization in a program has a lot to do with how much an application can be sped up. This is Amdahl's Law
** Amdahl's Law states that a parallel program can only be sped up by the inverse of the proportion of the program that cannot be made parallel (e.g. 25%(.25) non-parallel --> limit of 4x speedup) (I can't get this to sound right someone fix it please -[[Rannath]] <- I will fix [[Daniel B.]]
*Types of serializing interactions found in the MOSBENCH apps:
**Locking of shared data structure as the number of cores increase leads to an increase in lock wait time
**Writing to shared memory as the number of cores increase leads to an increase in the execution time of the cache coherence protocol
**Competing for space in shared hardware cache as the number of cores increase leads to an increase in cache miss rate
**Competing for shared hardware resources as the number of cores increase leads to time lost waiting for resources
**Not enough tasks for cores leads to idle cores

===Multicore packet processing: ''Section 4.2''===
Linux packet processing technique requires the packets to travel along several queues before it finally becomes available for the application to use. This technique works well for most general socket applications. In recent kernel releases Linux takes advantage of multiple hardware queues (when available on the given network interface) or Receive Packet Steering[1] to direct packet flow onto different cores for processing. Or even go as far as directing packet flow to the core on which the application is running using Receive Flow Steering[2] for even better performance. Linux also attempts to increase performance using a sampling technique where it checks every 20th outgoing packet and directs flow based on its hash. This poses a problem for short lived connections like those associated with Apache since there is great potential for packets to be misdirected.

In general this technique performs poorly when there are numerous open connections spread across multiple cores due to mutex (mutual exclusion) delays and cache misses. In such scenarios its better to process all connections, with associated packets and queues, on one core to avoid said issues. The patched kernel's implementation proposed in this article uses multiple hardware queues (which can be accomplished through Receive Packet Sharing) to direct all packets from a given connection to the same core. In turn Apache is modified to only accept connections if the thread dedicated to processing them is on the same core. If the current core's queue is found to be empty it will attempt to obtain work from queues located on different cores. This configuration is ideal for numerous short connections as all the work for them in accomplished quickly on one core avoiding unnecessary mutex delays associated with packet queues and inter-core cache misses.

[1] J. Corbet. Receive Packet Steering, November 2009. http://lwn.net/Articles/362339/.
[2] J. Edge. Receive Flow Steering, April 2010. http://lwn.net/Articles/382428/.

===Sloppy counters: ''Section 4.3''===
Bottlenecks were encountered when the applications undergoing testing were referencing and updating shared counters for multiple cores. The solution in the paper is to use sloppy counters, which gets each core to track its own separate counts of references and uses a central shared counter to keep all counts on track. This is ideal because each core updates its counts by modifying its per-core counter, usually only needing access to its own local cache, cutting down on waiting for locks or serialization. Sloppy counters are also backwards-compatible with existing shared-counter code, making its implementation much easier to accomplish. The main disadvantages of the sloppy counters are that in situations where object de-allocation occurs often, because the de-allocation itself is an expensive operation, and the counters use up space proportional to the number of cores.

===Lock-free comparison: ''Section 4.4''===
This section describes a specific instance of unnecessary locking.

===Per-Core Data Structures: ''Section 4.5''===
Three centralized data structures were causing bottlenecks - a per-superblock list of open files, vfsmount table, the packet buffers free list. Each data structure was decentralized to per-core versions of itself. In the case of vfsmount the central data structure was maintained, and any per-core misses got written from the central table to the per-core table.

===Eliminating false sharing: ''Section 4.6''===
Misplaced variables on the cache cause different cores to request the same line to be read and written at the same time often enough to significantly impact performance. By moving the often written variable to another line the bottleneck was removed.

===Avoiding unnecessary locking: ''Section 4.7''===
Many locks/mutexes have special cases where they don't need to lock. Likewise mutexes can be split from locking the whole data structure to locking a part of it. Both these changes remove or reduce bottlenecks.

==Conclusion: ''Sections 6 & 7''==
===Work in Progress===

====[[Rovic P.]]====
-I'm just using this as a notepad, do not copy/paste this section, I will put in a properly written set of paragraphs which will fit with the contribution questions asked. -RP

This research contributes by evaluating the scalability discrepancies of applications programming and kernel programming. Key discoveries in this research show the effectiveness of the kernel in handling scaling amongst CPU cores. This has also shown that scaling in application programming should be more the focus. It has been shown that simple scaling techniques (list techniques) such as programming parallelism (look up more stuff to back this up and quotes). (Sloppy counter effectiveness, possible positive contributions, what has been used (internet search), what hasn’t been used.) Read conclusion, 2nd paragraph.

One reason the
required changes are modest is that stock Linux already
incorporates many modifications to improve scalability.
More speculatively, perhaps it is the case that Linux’s
system-call API is well suited to an implementation that
avoids unnecessary contention over kernel objects.

====[[Rannath]]====
Everything so far indicates that the MOSBENCH applications can scale to 48 cores. This scaling required a few modest changes to remove bottlenecks. The MIT team speculate that that trend will continue as the number of cores increase. They also state that things not bottlenecked by the CPU are harder to fix.

We can eliminate most kernel bottlenecks that the applications hits most often with minor changes. Most changes were well known methodology, with the exception of Sloppy counters. This study is limited by the removal of the IO bottleneck, but it does suggest that traditional implementations can be made scalable.

==Critique==
What is good and not-so-good about this paper? You may discuss both the style and content;
be sure to ground your discussion with specific references. Simple assertions that something is good or bad is not enough - you must explain why.
Since this is a "my implementation is better then your implementation" paper the "goodness" of content can be impartially determined by its fairness and the honesty of the authors.

===Content(Fairness): ''Section 5''===

====memcached: ''Section 5.3''====
memcached is treated with near perfect fairness in the paper. Its an in-memory service, so the ignored storage IO bottleneck does not affect it at all. Likewise the "stock" and "PK" implementations are given the same test suite, so there is no advantage given to either. memcached itself is non-scalable, so the MIT team was forced to run one instance per-core to keep up throughput. The FAQ at memcached.org's wiki suggests using multiple implementations per-server as a work around to another problem, which implies that running multiple instances of the server is the same, or nearly the same, as running one larger server [3]. In the end memcached was bottlenecked by the network card.

====Apache: ''Section 5.4''====
Linux has a built in kernel flaw where network packets are forced to travel though multiple queues before they arrive at queue where they can be processed by the application. This imposes significant costs on multi-core systems due to queue locking costs. This flaw inherently diminishes the performance of Apache on multi-core system due to multiple threads spread across cores being forced to deal with these mutex (mutual exclusion) costs. For the sake of this experiment Apache had a separate instance on every core listening on different ports which is not a practical real world application but merely an attempt to implement better parallel execution on a traditional kernel. The patched kernel implementation of the network stack is also specific to the problem at hand, which is processing multiple short lived connections across multiple cores. Although this provides a performance increase in the given scenario, in more general applications network performance might suffer. These tests were also rigged to avoid bottlenecks in place by network and file storage hardware. Meaning, making the proposed modifications to the kernel wont necessarily produce the same increase in productivity as described in the article. This is very much evident in the test where performance degrades past 36 cores due to limitation of the networking hardware.

''Which is not a problem as the paper specifically states that they are testing what they can improve in spite of hardware limitation.'' - [[Rannath]]

====gmake: ''Section 5.6''====
Since the inherent nature of gmake makes it quite parallel, the testing and updating that were attempted on gmake resulted in essentially the same scalability results for both the stock and modified kernel. The only change that was found was that gmake spent slightly less time at the system level because of the changes that were made to the system's caching. As stated in the paper, the execution time of gmake relies quite heavily on the compiler that is uses with gmake, so depending on which compiler was chosen, gmake could run worse or even slightly better. In any case, there seems to be no fairness concerns when it comes to the scalability testing of gmake as the same application load-out was used for all of the tests.

====Conclusion: ''Sections 6 & 7''====
Given that all tests are more or less fair for the purposes of the benchmarks. They would support the Hypothesis that Linux can be made to scale, at least to 48 cores. Thus the conclusion is fair iff the rest of the paper is fair.

Now you just have to fill in how fair the rest of the paper is.

===Style===
Style Criterion (feel free to add I have no idea what should go here):
- does the paper present information out of order?
- does the paper present needless information?
- does the paper have any sections that are inherently confusing? Wrong?
- is the paper easy to read through, or does it change subjects repeatedly?
- does the paper have too many "long-winded" sentences, making it seem like the authors are just trying to add extra words to make it seem more important? - I think maybe limit this to run-on sentences.
- Check for grammar

Everything seems to be in logical order. I couldn't find any needless info. Nothing inherently confusing or wrong. Nothing bad on the grammar front either. - Rannath

Some acronyms aren't explained before they are used, so some people reading the paper may get confused as to what they mean (e.g. Linux TLB). Since this paper is meant to be formal, acronyms should be explained, with some exceptions like OS and IBM. - Daniel B.

Your example has no impact on the paper, it was in the "look here for more info" section. Most people wouldn't know what a "translation look-aside buffer" is either.

==References==

[3] memcached's wiki: http://code.google.com/p/memcached/wiki/FAQ#Can_I_use_different_size_caches_across_servers_and_will_memcache

You will almost certainly have to refer to other resources; please cite these resources in the style of citation of the papers assigned (inlined numbered references). Place your bibliographic entries in this section.

'''the paper itself doesn't need to be referenced more than once as this is a critique of the paper...'''
[1] Silas Boyd-Wickizer et al. "An Analysis of Linux Scalability to Many Cores". In ''OSDI '10, 9th USENIX Symposium on OS Design and Implementation'', Vancouver, BC, Canada, 2010. http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf.

gmake:

[http://www.gnu.org/software/make/manual/make.html gmake Manual]

[http://www.gnu.org/software/make/ gmake Main Page]

==Deprecated==
===Background Concepts===
* Exim: ''Section 3.1'':
**Exim is a mail server for Unix. It's fairly parallel. The server forks a new process for each connection and twice to deliver each message. It spends 69% of its time in the kernel on a single core.

* PostgreSQL: ''Section 3.4'':
**As implied by the name PostgreSQL is a SQL database. PostgreSQL starts a separate process for each connection and uses kernel locking interfaces extensively to provide concurrent access to the database. Due to bottlenecks introduced in its code and in the kernel code, the amount of time PostgreSQL spends in the kernel increases very rapidly with addition of new cores. On a single core system PostgreSQL spends only 1.5% of its time in the kernel. On a 48 core system the execution time in the kernel jumps to 82%.

Talk:COMP 3000 Essay 2 2010 Question 1

2010-12-02T17:26:09Z

Kkashigi: /* Multicore packet processing: Section 4.2 */

=Class and Notices=
(Nov. 30, 2010) Prof. Anil stated that we should focus on the 3 easiest to understand parts in section 5 and elaborate on them.

- Also, I, Daniel B., work Thursday night, so I will be finishing up as much of my part as I can for the essay before Thursday morning's class, then maybe we can all meet up in a lab in HP and put the finishing touches on the essay. I will be available online Wednesday night from about 6:30pm onwards and will be in the game dev lab or CCSS lounge Wednesday morning from about 11am to 2pm if anyone would like to meet up with me at those times.

- [[I suggest we meet up Thursday morning after Operating Systems in order to discuss and finalize the essay. Maybe we can even designate a lab for the group to meet up in. Any suggestions?]] - Daniel B.

- HP 3115 since there wont be a class in there (as its our tutorial and we know there won't be anyone there)
-- Go to Wireless Lab next to CCSS Lounge. Andrew and Dan B. will be there.

- If its all the same to you guys mind if I just join you via msn or iirc? Or phone if you really want. -Rannath

- I'm working today, but I'll be at a computer reading this page/contributing to my section. Depending on how busy I am, I should be able to get some significant writing in before 4pm today on my section and any additional sections required. RP

- I wont be there either. that does not mean i wont/cant contribute. I'll be on msn or you can just email me. -kirill

=Group members=
Patrick Young [mailto:Rannath@gmail.com Rannath@gmail.com]

Daniel Beimers [mailto:demongyro@gmail.com demongyro@gmail.com]

Andrew Bown [mailto:abown2@connect.carleton.ca abown2@connect.carleton.ca]

Kirill Kashigin [mailto:k.kashigin@gmail.com k.kashigin@gmail.com]

Rovic Perdon [mailto:rperdon@gmail.com rperdon@gmail.com]

Daniel Sont [mailto:dan.sont@gmail.com dan.sont@gmail.com]

=Methodology=
We should probably have our work verified by at least one group member before posting to the actual page

=To Do=
*Improve the grammar/structure of the paper section & add links to supplementary info
*flesh out the whole lot

===Claim Sections===
* So here is the claims and unclaimed section. Add your name next to one if you want to take it on.
** gmake - Daniel B.
** memcached - Rannath
** Apache - Kirill
** [[(Exim, PostgreSQL, Metis, and Psearchy will not be needed as the professor said we only need to explain 3)]]
** Research Problem - Andrew
** Contribution - Rovic
** Essay Conclusion (also discussion) - Everyone
** Critic, Style - Everyone
** References - Everyone

=Essay=
==Paper - DONE!!!==
This paper was authored by - Silas Boyd-Wickizer, Austin T. Clements, Yandong Mao, Aleksey Pesterev, M. Frans Kaashoek, Robert Morris, and Nickolai Zeldovich.

They all work at MIT CSAIL.

The paper: [http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf An Analysis of Linux Scalability to Many Cores]

==Background Concepts==
Explain briefly the background concepts and ideas that your fellow classmates will need to know first in order to understand your assigned paper.

===memcached: ''Section 3.2''===
memcached is an in-memory hash table server. One instance of memcached running on many different cores is bottlenecked by an internal lock, which is avoided by the MIT team by running one instance per core. Clients each connect to a single instance of memcached, allowing the server to simulate parallelism without needing to make major changes to the application or kernel. With few requests, memcached spends 80% of its time in the kernel on one core, mostly processing packets. [1]

===Apache: ''Section 3.3''===
Apache is a web server. In the case of this study, Apache has been configured to run a separate process on each core. Each process, in turn, has multiple threads (Making it a perfect example of parallel programming). One thread to service incoming connections and various other threads to service those connections. On a single core processor, Apache spends 60% of its execution time in the kernel.

===gmake: ''Section 3.5''===
gmake is an unofficial default benchmark in the Linux community which is used in this paper to build the Linux kernel. gmake takes a file called a makefile and processes its recipes for the requisite files to determine how and when to remake or recompile code. With a simple command -j or --jobs, gmake can process many of these recipes in parallel. Since gmake creates more processes than cores, it can make proper use of multiple cores to process the recipes.[2] Since gmake involves much reading and writing, in order to prevent bottlenecks due to the filesystem or hardware, the test cases use an in-memory filesystem tmpfs, which gives them a backdoor around the bottlenecks for testing purposes. In addition to this, gmake is limited in scalability due to the serial processes that run at the beginning and end of its execution, which limits its scalability to a small degree. gmake spends much of its execution time with its compiler, processing the recipes and recompiling code, but still spend 7.6% of its time in system time.[1]

[2] http://www.gnu.org/software/make/manual/make.html

==Research problem==
my references are just below because it is easier for numbering the data later.

As technological progress the number of core a main processor can have is increasing at an impressive rate. Soon personal computers will have so many cores that scalability will be an issue. There has to be a way that standard user level Linux kernel will scale with a 48-core system[1]. The problem with a standard Linux operating is they are not designed for massive scalability which will soon be a problem. The issue with scalability is that a solo core will perform much more work compared to a single core working with 47 other cores. Although traditional logic that situation makes sense because 48 cores are dividing the work. But when processing information a process the main goal is to finish so as long as possible every core should be doing a much work as possible.

To fix those scalability issues it is necessary to focus on three major areas: the Linux kernel, user level design and how application use of kernel services. The Linux kernel can be improved be to improve sharing and have the advantage of recent iterations are beginning to implement scalability features. On the user level design applications can be improved so that there is more focus on parallelism since some programs have not implements those improved features. The final aspect of improving scalability is how an application uses kernel services to share resources better so that different aspects of the program are not conflicting over the same services. All of the bottlenecks are found actually only take a little work to avoid.[1]

This research is based on much research which was created before in the development of scalability for UNIX system. The major developments from shared memory machines [2], wait-free synchronization to fast message passing have created a base set of techniques which can be used to improve scalability. These techniques have been incorporated in all major operation system including Linux, Mac OS X and Windows. Linux has been improved with kernel subsystems such as Read-Copy-Update which an algorithm for which is used to avoid locks and atomic instructions which lower scalability.[3] The is also an excellent base a research on Linux scalability studies to base this research paper. These paper include a on doing scalability on a 32-core machine. [4] That research can improve the results by learning from the experiments already performed by researchers. This research also aid identifying bottlenecks which speed up researching solutions for those bottlenecks.

[2] J. Kuskin, D. Ofelt, M. Heinrich, J. Heinlein, R. Simoni, K. Gharachorloo, J. Chapin, D. Nakahira, J. Baxter, M. Horowitz, A. Gupta, M. Rosenblum, and J. Hennessy. The Stanford FLASH multiprocessor. In Proc. of the 21st ISCA, pages 302–313,1994.

[3] P. E. McKenney, D. Sarma, A. Arcangeli, A. Kleen, O. Krieger, and R. Russell. Read-copy-update. In Proceedings of the Linux Symposium 2002, pages 338-367, Ottawa Ontario, June 2002

[4] C. Yan, Y. Chen, and S. Yuanchun. OSMark: A benchmark suite for understanding parallel scalability of operating systems on large scale multi-cores. In 2009 2nd International Conference on Computer Science and Information Technology, pages 313–317, 2009

==Contribution==
What was implemented? Why is it any better than what came before?

Summarize info from Section 4.2 onwards, maybe put graphs from Section 5 here to provide support for improvements (if that isn't unethical/illegal)?

- So long as we cite the paper and don't pretend the graphs are ours, we are ok, since we are writing an explanation/critic of the paper.

All contributions in this paper are the result of the identification and removal or marginalization of bottlenecks.

===What hinders scalability: ''Section 4.1''===
*The percentage of serialization in a program has a lot to do with how much an application can be sped up. This is Amdahl's Law
** Amdahl's Law states that a parallel program can only be sped up by the inverse of the proportion of the program that cannot be made parallel (e.g. 25%(.25) non-parallel --> limit of 4x speedup) (I can't get this to sound right someone fix it please -[[Rannath]] <- I will fix [[Daniel B.]]
*Types of serializing interactions found in the MOSBENCH apps:
**Locking of shared data structure as the number of cores increase leads to an increase in lock wait time
**Writing to shared memory as the number of cores increase leads to an increase in the execution time of the cache coherence protocol
**Competing for space in shared hardware cache as the number of cores increase leads to an increase in cache miss rate
**Competing for shared hardware resources as the number of cores increase leads to time lost waiting for resources
**Not enough tasks for cores leads to idle cores

===Multicore packet processing: ''Section 4.2''===
Linux packet processing technique requires the packets to travel along several queues before it finally becomes available for the application to use. This technique works well for most general socket applications. In recent kernel releases Linux takes advantage of multiple hardware queues (when available on the given network interface) or Receive Packet Steering[1] to direct packet flow onto different cores for processing. Or even go as far as directing packet flow to the core on which the application is running using Receive Flow Steering[2] for even better performance. Linux also attempts to increase performance using a sampling technique where it checks every 20th outgoing packet and directs flow based on its hash. This poses a problem for short lived connections like those associated with Apache since there is great potential for packets to be misdirected.

In general this technique performs poorly when there are numerous open connections spread across multiple cores due to mutex (mutual exclusion) delays and cache misses. In such scenarios its better to process all connections, with associated packets and queues, on one core to avoid said issues. The patched kernel's implementation proposed in this article uses multiple hardware queues (which can be accomplished through Receive Packet Sharing) to direct all packets from a given connection to the same core. In turn Apache is modified to only accept connections if the thread dedicated to processing them is on the same core. If the current core's queue is found to be empty it will attempt to obtain work from queues located on different cores. This configuration is ideal for numerous short connections as all the work for them in accomplished quickly on one core avoiding unnecessary mutex delays associated with packet queues and inter-core cache misses.

===Sloppy counters: ''Section 4.3''===
Bottlenecks were encountered when the applications undergoing testing were referencing and updating shared counters for multiple cores. The solution in the paper is to use sloppy counters, which gets each core to track its own separate counts of references and uses a central shared counter to keep all counts on track. This is ideal because each core updates its counts by modifying its per-core counter, usually only needing access to its own local cache, cutting down on waiting for locks or serialization. Sloppy counters are also backwards-compatible with existing shared-counter code, making its implementation much easier to accomplish. The main disadvantages of the sloppy counters are that in situations where object de-allocation occurs often, because the de-allocation itself is an expensive operation, and the counters use up space proportional to the number of cores.

===Lock-free comparison: ''Section 4.4''===
This section describes a specific instance of unnecessary locking.

===Per-Core Data Structures: ''Section 4.5''===
Three centralized data structures were causing bottlenecks - a per-superblock list of open files, vfsmount table, the packet buffers free list. Each data structure was decentralized to per-core versions of itself. In the case of vfsmount the central data structure was maintained, and any per-core misses got written from the central table to the per-core table.

===Eliminating false sharing: ''Section 4.6''===
Misplaced variables on the cache cause different cores to request the same line to be read and written at the same time often enough to significantly impact performance. By moving the often written variable to another line the bottleneck was removed.

===Avoiding unnecessary locking: ''Section 4.7''===
Many locks/mutexes have special cases where they don't need to lock. Likewise mutexes can be split from locking the whole data structure to locking a part of it. Both these changes remove or reduce bottlenecks.

==Conclusion: ''Sections 6 & 7''==
===Work in Progress===

====[[Rovic P.]]====
-I'm just using this as a notepad, do not copy/paste this section, I will put in a properly written set of paragraphs which will fit with the contribution questions asked. -RP

This research contributes by evaluating the scalability discrepancies of applications programming and kernel programming. Key discoveries in this research show the effectiveness of the kernel in handling scaling amongst CPU cores. This has also shown that scaling in application programming should be more the focus. It has been shown that simple scaling techniques (list techniques) such as programming parallelism (look up more stuff to back this up and quotes). (Sloppy counter effectiveness, possible positive contributions, what has been used (internet search), what hasn’t been used.) Read conclusion, 2nd paragraph.

One reason the
required changes are modest is that stock Linux already
incorporates many modifications to improve scalability.
More speculatively, perhaps it is the case that Linux’s
system-call API is well suited to an implementation that
avoids unnecessary contention over kernel objects.

====[[Rannath]]====
Everything so far indicates that the MOSBENCH applications can scale to 48 cores. This scaling required a few modest changes to remove bottlenecks. The MIT team speculate that that trend will continue as the number of cores increase. They also state that things not bottlenecked by the CPU are harder to fix.

We can eliminate most kernel bottlenecks that the applications hits most often with minor changes. Most changes were well known methodology, with the exception of Sloppy counters. This study is limited by the removal of the IO bottleneck, but it does suggest that traditional implementations can be made scalable.

==Critique==
What is good and not-so-good about this paper? You may discuss both the style and content;
be sure to ground your discussion with specific references. Simple assertions that something is good or bad is not enough - you must explain why.
Since this is a "my implementation is better then your implementation" paper the "goodness" of content can be impartially determined by its fairness and the honesty of the authors.

===Content(Fairness): ''Section 5''===

====memcached: ''Section 5.3''====
memcached is treated with near perfect fairness in the paper. Its an in-memory service, so the ignored storage IO bottleneck does not affect it at all. Likewise the "stock" and "PK" implementations are given the same test suite, so there is no advantage given to either. memcached itself is non-scalable, so the MIT team was forced to run one instance per-core to keep up throughput. The FAQ at memcached.org's wiki suggests using multiple implementations per-server as a work around to another problem, which implies that running multiple instances of the server is the same, or nearly the same, as running one larger server [3]. In the end memcached was bottlenecked by the network card.

====Apache: ''Section 5.4''====
Linux has a built in kernel flaw where network packets are forced to travel though multiple queues before they arrive at queue where they can be processed by the application. This imposes significant costs on multi-core systems due to queue locking costs. This flaw inherently diminishes the performance of Apache on multi-core system due to multiple threads spread across cores being forced to deal with these mutex (mutual exclusion) costs. For the sake of this experiment Apache had a separate instance on every core listening on different ports which is not a practical real world application but merely an attempt to implement better parallel execution on a traditional kernel. The patched kernel implementation of the network stack is also specific to the problem at hand, which is processing multiple short lived connections across multiple cores. Although this provides a performance increase in the given scenario, in more general applications network performance might suffer. These tests were also rigged to avoid bottlenecks in place by network and file storage hardware. Meaning, making the proposed modifications to the kernel wont necessarily produce the same increase in productivity as described in the article. This is very much evident in the test where performance degrades past 36 cores due to limitation of the networking hardware.

''Which is not a problem as the paper specifically states that they are testing what they can improve in spite of hardware limitation.'' - [[Rannath]]

====gmake: ''Section 5.6''====
Since the inherent nature of gmake makes it quite parallel, the testing and updating that were attempted on gmake resulted in essentially the same scalability results for both the stock and modified kernel. The only change that was found was that gmake spent slightly less time at the system level because of the changes that were made to the system's caching. As stated in the paper, the execution time of gmake relies quite heavily on the compiler that is uses with gmake, so depending on which compiler was chosen, gmake could run worse or even slightly better. In any case, there seems to be no fairness concerns when it comes to the scalability testing of gmake as the same application load-out was used for all of the tests.

====Conclusion: ''Sections 6 & 7''====
Given that all tests are more or less fair for the purposes of the benchmarks. They would support the Hypothesis that Linux can be made to scale, at least to 48 cores. Thus the conclusion is fair iff the rest of the paper is fair.

Now you just have to fill in how fair the rest of the paper is.

===Style===
Style Criterion (feel free to add I have no idea what should go here):
- does the paper present information out of order?
- does the paper present needless information?
- does the paper have any sections that are inherently confusing? Wrong?
- is the paper easy to read through, or does it change subjects repeatedly?
- does the paper have too many "long-winded" sentences, making it seem like the authors are just trying to add extra words to make it seem more important? - I think maybe limit this to run-on sentences.
- Check for grammar

Everything seems to be in logical order. I couldn't find any needless info. Nothing inherently confusing or wrong. Nothing bad on the grammar front either. - Rannath

Some acronyms aren't explained before they are used, so some people reading the paper may get confused as to what they mean (e.g. Linux TLB). Since this paper is meant to be formal, acronyms should be explained, with some exceptions like OS and IBM. - Daniel B.

Your example has no impact on the paper, it was in the "look here for more info" section. Most people wouldn't know what a "translation look-aside buffer" is either.

==References==

[3] memcached's wiki: http://code.google.com/p/memcached/wiki/FAQ#Can_I_use_different_size_caches_across_servers_and_will_memcache

You will almost certainly have to refer to other resources; please cite these resources in the style of citation of the papers assigned (inlined numbered references). Place your bibliographic entries in this section.

'''the paper itself doesn't need to be referenced more than once as this is a critique of the paper...'''
[1] Silas Boyd-Wickizer et al. "An Analysis of Linux Scalability to Many Cores". In ''OSDI '10, 9th USENIX Symposium on OS Design and Implementation'', Vancouver, BC, Canada, 2010. http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf.

gmake:

[http://www.gnu.org/software/make/manual/make.html gmake Manual]

[http://www.gnu.org/software/make/ gmake Main Page]

==Deprecated==
===Background Concepts===
* Exim: ''Section 3.1'':
**Exim is a mail server for Unix. It's fairly parallel. The server forks a new process for each connection and twice to deliver each message. It spends 69% of its time in the kernel on a single core.

* PostgreSQL: ''Section 3.4'':
**As implied by the name PostgreSQL is a SQL database. PostgreSQL starts a separate process for each connection and uses kernel locking interfaces extensively to provide concurrent access to the database. Due to bottlenecks introduced in its code and in the kernel code, the amount of time PostgreSQL spends in the kernel increases very rapidly with addition of new cores. On a single core system PostgreSQL spends only 1.5% of its time in the kernel. On a 48 core system the execution time in the kernel jumps to 82%.

Talk:COMP 3000 Essay 2 2010 Question 1

2010-12-02T17:25:56Z

Kkashigi: /* Multicore packet processing: Section 4.2 */

=Class and Notices=
(Nov. 30, 2010) Prof. Anil stated that we should focus on the 3 easiest to understand parts in section 5 and elaborate on them.

- Also, I, Daniel B., work Thursday night, so I will be finishing up as much of my part as I can for the essay before Thursday morning's class, then maybe we can all meet up in a lab in HP and put the finishing touches on the essay. I will be available online Wednesday night from about 6:30pm onwards and will be in the game dev lab or CCSS lounge Wednesday morning from about 11am to 2pm if anyone would like to meet up with me at those times.

- [[I suggest we meet up Thursday morning after Operating Systems in order to discuss and finalize the essay. Maybe we can even designate a lab for the group to meet up in. Any suggestions?]] - Daniel B.

- HP 3115 since there wont be a class in there (as its our tutorial and we know there won't be anyone there)
-- Go to Wireless Lab next to CCSS Lounge. Andrew and Dan B. will be there.

- If its all the same to you guys mind if I just join you via msn or iirc? Or phone if you really want. -Rannath

- I'm working today, but I'll be at a computer reading this page/contributing to my section. Depending on how busy I am, I should be able to get some significant writing in before 4pm today on my section and any additional sections required. RP

- I wont be there either. that does not mean i wont/cant contribute. I'll be on msn or you can just email me. -kirill

=Group members=
Patrick Young [mailto:Rannath@gmail.com Rannath@gmail.com]

Daniel Beimers [mailto:demongyro@gmail.com demongyro@gmail.com]

Andrew Bown [mailto:abown2@connect.carleton.ca abown2@connect.carleton.ca]

Kirill Kashigin [mailto:k.kashigin@gmail.com k.kashigin@gmail.com]

Rovic Perdon [mailto:rperdon@gmail.com rperdon@gmail.com]

Daniel Sont [mailto:dan.sont@gmail.com dan.sont@gmail.com]

=Methodology=
We should probably have our work verified by at least one group member before posting to the actual page

=To Do=
*Improve the grammar/structure of the paper section & add links to supplementary info
*flesh out the whole lot

===Claim Sections===
* So here is the claims and unclaimed section. Add your name next to one if you want to take it on.
** gmake - Daniel B.
** memcached - Rannath
** Apache - Kirill
** [[(Exim, PostgreSQL, Metis, and Psearchy will not be needed as the professor said we only need to explain 3)]]
** Research Problem - Andrew
** Contribution - Rovic
** Essay Conclusion (also discussion) - Everyone
** Critic, Style - Everyone
** References - Everyone

=Essay=
==Paper - DONE!!!==
This paper was authored by - Silas Boyd-Wickizer, Austin T. Clements, Yandong Mao, Aleksey Pesterev, M. Frans Kaashoek, Robert Morris, and Nickolai Zeldovich.

They all work at MIT CSAIL.

The paper: [http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf An Analysis of Linux Scalability to Many Cores]

==Background Concepts==
Explain briefly the background concepts and ideas that your fellow classmates will need to know first in order to understand your assigned paper.

===memcached: ''Section 3.2''===
memcached is an in-memory hash table server. One instance of memcached running on many different cores is bottlenecked by an internal lock, which is avoided by the MIT team by running one instance per core. Clients each connect to a single instance of memcached, allowing the server to simulate parallelism without needing to make major changes to the application or kernel. With few requests, memcached spends 80% of its time in the kernel on one core, mostly processing packets. [1]

===Apache: ''Section 3.3''===
Apache is a web server. In the case of this study, Apache has been configured to run a separate process on each core. Each process, in turn, has multiple threads (Making it a perfect example of parallel programming). One thread to service incoming connections and various other threads to service those connections. On a single core processor, Apache spends 60% of its execution time in the kernel.

===gmake: ''Section 3.5''===
gmake is an unofficial default benchmark in the Linux community which is used in this paper to build the Linux kernel. gmake takes a file called a makefile and processes its recipes for the requisite files to determine how and when to remake or recompile code. With a simple command -j or --jobs, gmake can process many of these recipes in parallel. Since gmake creates more processes than cores, it can make proper use of multiple cores to process the recipes.[2] Since gmake involves much reading and writing, in order to prevent bottlenecks due to the filesystem or hardware, the test cases use an in-memory filesystem tmpfs, which gives them a backdoor around the bottlenecks for testing purposes. In addition to this, gmake is limited in scalability due to the serial processes that run at the beginning and end of its execution, which limits its scalability to a small degree. gmake spends much of its execution time with its compiler, processing the recipes and recompiling code, but still spend 7.6% of its time in system time.[1]

[2] http://www.gnu.org/software/make/manual/make.html

==Research problem==
my references are just below because it is easier for numbering the data later.

As technological progress the number of core a main processor can have is increasing at an impressive rate. Soon personal computers will have so many cores that scalability will be an issue. There has to be a way that standard user level Linux kernel will scale with a 48-core system[1]. The problem with a standard Linux operating is they are not designed for massive scalability which will soon be a problem. The issue with scalability is that a solo core will perform much more work compared to a single core working with 47 other cores. Although traditional logic that situation makes sense because 48 cores are dividing the work. But when processing information a process the main goal is to finish so as long as possible every core should be doing a much work as possible.

To fix those scalability issues it is necessary to focus on three major areas: the Linux kernel, user level design and how application use of kernel services. The Linux kernel can be improved be to improve sharing and have the advantage of recent iterations are beginning to implement scalability features. On the user level design applications can be improved so that there is more focus on parallelism since some programs have not implements those improved features. The final aspect of improving scalability is how an application uses kernel services to share resources better so that different aspects of the program are not conflicting over the same services. All of the bottlenecks are found actually only take a little work to avoid.[1]

This research is based on much research which was created before in the development of scalability for UNIX system. The major developments from shared memory machines [2], wait-free synchronization to fast message passing have created a base set of techniques which can be used to improve scalability. These techniques have been incorporated in all major operation system including Linux, Mac OS X and Windows. Linux has been improved with kernel subsystems such as Read-Copy-Update which an algorithm for which is used to avoid locks and atomic instructions which lower scalability.[3] The is also an excellent base a research on Linux scalability studies to base this research paper. These paper include a on doing scalability on a 32-core machine. [4] That research can improve the results by learning from the experiments already performed by researchers. This research also aid identifying bottlenecks which speed up researching solutions for those bottlenecks.

[2] J. Kuskin, D. Ofelt, M. Heinrich, J. Heinlein, R. Simoni, K. Gharachorloo, J. Chapin, D. Nakahira, J. Baxter, M. Horowitz, A. Gupta, M. Rosenblum, and J. Hennessy. The Stanford FLASH multiprocessor. In Proc. of the 21st ISCA, pages 302–313,1994.

[3] P. E. McKenney, D. Sarma, A. Arcangeli, A. Kleen, O. Krieger, and R. Russell. Read-copy-update. In Proceedings of the Linux Symposium 2002, pages 338-367, Ottawa Ontario, June 2002

[4] C. Yan, Y. Chen, and S. Yuanchun. OSMark: A benchmark suite for understanding parallel scalability of operating systems on large scale multi-cores. In 2009 2nd International Conference on Computer Science and Information Technology, pages 313–317, 2009

==Contribution==
What was implemented? Why is it any better than what came before?

Summarize info from Section 4.2 onwards, maybe put graphs from Section 5 here to provide support for improvements (if that isn't unethical/illegal)?

- So long as we cite the paper and don't pretend the graphs are ours, we are ok, since we are writing an explanation/critic of the paper.

All contributions in this paper are the result of the identification and removal or marginalization of bottlenecks.

===What hinders scalability: ''Section 4.1''===
*The percentage of serialization in a program has a lot to do with how much an application can be sped up. This is Amdahl's Law
** Amdahl's Law states that a parallel program can only be sped up by the inverse of the proportion of the program that cannot be made parallel (e.g. 25%(.25) non-parallel --> limit of 4x speedup) (I can't get this to sound right someone fix it please -[[Rannath]] <- I will fix [[Daniel B.]]
*Types of serializing interactions found in the MOSBENCH apps:
**Locking of shared data structure as the number of cores increase leads to an increase in lock wait time
**Writing to shared memory as the number of cores increase leads to an increase in the execution time of the cache coherence protocol
**Competing for space in shared hardware cache as the number of cores increase leads to an increase in cache miss rate
**Competing for shared hardware resources as the number of cores increase leads to time lost waiting for resources
**Not enough tasks for cores leads to idle cores

===Multicore packet processing: ''Section 4.2''===
Linux packet processing technique requires the packets to travel along several queues before it finally becomes available for the application to use. This technique works well for most general socket applications. In recent kernel releases Linux takes advantage of multiple hardware queues (when available on the given network interface) or Receive Packet Steering[1] to direct packet flow onto different cores for processing. Or even go as far as directing packet flow to the core on which the application is running using Receive Flow Steering[2] for even better performance. Linux also attempts to increase performance using a sampling technique where it checks every 20th outgoing packet and directs flow based on its hash. This poses a problem for short lived connections like those associated with Apache since there is great potential for packets to be misdirected.
In general this technique performs poorly when there are numerous open connections spread across multiple cores due to mutex (mutual exclusion) delays and cache misses. In such scenarios its better to process all connections, with associated packets and queues, on one core to avoid said issues. The patched kernel's implementation proposed in this article uses multiple hardware queues (which can be accomplished through Receive Packet Sharing) to direct all packets from a given connection to the same core. In turn Apache is modified to only accept connections if the thread dedicated to processing them is on the same core. If the current core's queue is found to be empty it will attempt to obtain work from queues located on different cores. This configuration is ideal for numerous short connections as all the work for them in accomplished quickly on one core avoiding unnecessary mutex delays associated with packet queues and inter-core cache misses.

===Sloppy counters: ''Section 4.3''===
Bottlenecks were encountered when the applications undergoing testing were referencing and updating shared counters for multiple cores. The solution in the paper is to use sloppy counters, which gets each core to track its own separate counts of references and uses a central shared counter to keep all counts on track. This is ideal because each core updates its counts by modifying its per-core counter, usually only needing access to its own local cache, cutting down on waiting for locks or serialization. Sloppy counters are also backwards-compatible with existing shared-counter code, making its implementation much easier to accomplish. The main disadvantages of the sloppy counters are that in situations where object de-allocation occurs often, because the de-allocation itself is an expensive operation, and the counters use up space proportional to the number of cores.

===Lock-free comparison: ''Section 4.4''===
This section describes a specific instance of unnecessary locking.

===Per-Core Data Structures: ''Section 4.5''===
Three centralized data structures were causing bottlenecks - a per-superblock list of open files, vfsmount table, the packet buffers free list. Each data structure was decentralized to per-core versions of itself. In the case of vfsmount the central data structure was maintained, and any per-core misses got written from the central table to the per-core table.

===Eliminating false sharing: ''Section 4.6''===
Misplaced variables on the cache cause different cores to request the same line to be read and written at the same time often enough to significantly impact performance. By moving the often written variable to another line the bottleneck was removed.

===Avoiding unnecessary locking: ''Section 4.7''===
Many locks/mutexes have special cases where they don't need to lock. Likewise mutexes can be split from locking the whole data structure to locking a part of it. Both these changes remove or reduce bottlenecks.

==Conclusion: ''Sections 6 & 7''==
===Work in Progress===

====[[Rovic P.]]====
-I'm just using this as a notepad, do not copy/paste this section, I will put in a properly written set of paragraphs which will fit with the contribution questions asked. -RP

This research contributes by evaluating the scalability discrepancies of applications programming and kernel programming. Key discoveries in this research show the effectiveness of the kernel in handling scaling amongst CPU cores. This has also shown that scaling in application programming should be more the focus. It has been shown that simple scaling techniques (list techniques) such as programming parallelism (look up more stuff to back this up and quotes). (Sloppy counter effectiveness, possible positive contributions, what has been used (internet search), what hasn’t been used.) Read conclusion, 2nd paragraph.

One reason the
required changes are modest is that stock Linux already
incorporates many modifications to improve scalability.
More speculatively, perhaps it is the case that Linux’s
system-call API is well suited to an implementation that
avoids unnecessary contention over kernel objects.

====[[Rannath]]====
Everything so far indicates that the MOSBENCH applications can scale to 48 cores. This scaling required a few modest changes to remove bottlenecks. The MIT team speculate that that trend will continue as the number of cores increase. They also state that things not bottlenecked by the CPU are harder to fix.

We can eliminate most kernel bottlenecks that the applications hits most often with minor changes. Most changes were well known methodology, with the exception of Sloppy counters. This study is limited by the removal of the IO bottleneck, but it does suggest that traditional implementations can be made scalable.

==Critique==
What is good and not-so-good about this paper? You may discuss both the style and content;
be sure to ground your discussion with specific references. Simple assertions that something is good or bad is not enough - you must explain why.
Since this is a "my implementation is better then your implementation" paper the "goodness" of content can be impartially determined by its fairness and the honesty of the authors.

===Content(Fairness): ''Section 5''===

====memcached: ''Section 5.3''====
memcached is treated with near perfect fairness in the paper. Its an in-memory service, so the ignored storage IO bottleneck does not affect it at all. Likewise the "stock" and "PK" implementations are given the same test suite, so there is no advantage given to either. memcached itself is non-scalable, so the MIT team was forced to run one instance per-core to keep up throughput. The FAQ at memcached.org's wiki suggests using multiple implementations per-server as a work around to another problem, which implies that running multiple instances of the server is the same, or nearly the same, as running one larger server [3]. In the end memcached was bottlenecked by the network card.

====Apache: ''Section 5.4''====
Linux has a built in kernel flaw where network packets are forced to travel though multiple queues before they arrive at queue where they can be processed by the application. This imposes significant costs on multi-core systems due to queue locking costs. This flaw inherently diminishes the performance of Apache on multi-core system due to multiple threads spread across cores being forced to deal with these mutex (mutual exclusion) costs. For the sake of this experiment Apache had a separate instance on every core listening on different ports which is not a practical real world application but merely an attempt to implement better parallel execution on a traditional kernel. The patched kernel implementation of the network stack is also specific to the problem at hand, which is processing multiple short lived connections across multiple cores. Although this provides a performance increase in the given scenario, in more general applications network performance might suffer. These tests were also rigged to avoid bottlenecks in place by network and file storage hardware. Meaning, making the proposed modifications to the kernel wont necessarily produce the same increase in productivity as described in the article. This is very much evident in the test where performance degrades past 36 cores due to limitation of the networking hardware.

''Which is not a problem as the paper specifically states that they are testing what they can improve in spite of hardware limitation.'' - [[Rannath]]

====gmake: ''Section 5.6''====
Since the inherent nature of gmake makes it quite parallel, the testing and updating that were attempted on gmake resulted in essentially the same scalability results for both the stock and modified kernel. The only change that was found was that gmake spent slightly less time at the system level because of the changes that were made to the system's caching. As stated in the paper, the execution time of gmake relies quite heavily on the compiler that is uses with gmake, so depending on which compiler was chosen, gmake could run worse or even slightly better. In any case, there seems to be no fairness concerns when it comes to the scalability testing of gmake as the same application load-out was used for all of the tests.

====Conclusion: ''Sections 6 & 7''====
Given that all tests are more or less fair for the purposes of the benchmarks. They would support the Hypothesis that Linux can be made to scale, at least to 48 cores. Thus the conclusion is fair iff the rest of the paper is fair.

Now you just have to fill in how fair the rest of the paper is.

===Style===
Style Criterion (feel free to add I have no idea what should go here):
- does the paper present information out of order?
- does the paper present needless information?
- does the paper have any sections that are inherently confusing? Wrong?
- is the paper easy to read through, or does it change subjects repeatedly?
- does the paper have too many "long-winded" sentences, making it seem like the authors are just trying to add extra words to make it seem more important? - I think maybe limit this to run-on sentences.
- Check for grammar

Everything seems to be in logical order. I couldn't find any needless info. Nothing inherently confusing or wrong. Nothing bad on the grammar front either. - Rannath

Some acronyms aren't explained before they are used, so some people reading the paper may get confused as to what they mean (e.g. Linux TLB). Since this paper is meant to be formal, acronyms should be explained, with some exceptions like OS and IBM. - Daniel B.

Your example has no impact on the paper, it was in the "look here for more info" section. Most people wouldn't know what a "translation look-aside buffer" is either.

==References==

[3] memcached's wiki: http://code.google.com/p/memcached/wiki/FAQ#Can_I_use_different_size_caches_across_servers_and_will_memcache

You will almost certainly have to refer to other resources; please cite these resources in the style of citation of the papers assigned (inlined numbered references). Place your bibliographic entries in this section.

'''the paper itself doesn't need to be referenced more than once as this is a critique of the paper...'''
[1] Silas Boyd-Wickizer et al. "An Analysis of Linux Scalability to Many Cores". In ''OSDI '10, 9th USENIX Symposium on OS Design and Implementation'', Vancouver, BC, Canada, 2010. http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf.

gmake:

[http://www.gnu.org/software/make/manual/make.html gmake Manual]

[http://www.gnu.org/software/make/ gmake Main Page]

==Deprecated==
===Background Concepts===
* Exim: ''Section 3.1'':
**Exim is a mail server for Unix. It's fairly parallel. The server forks a new process for each connection and twice to deliver each message. It spends 69% of its time in the kernel on a single core.

* PostgreSQL: ''Section 3.4'':
**As implied by the name PostgreSQL is a SQL database. PostgreSQL starts a separate process for each connection and uses kernel locking interfaces extensively to provide concurrent access to the database. Due to bottlenecks introduced in its code and in the kernel code, the amount of time PostgreSQL spends in the kernel increases very rapidly with addition of new cores. On a single core system PostgreSQL spends only 1.5% of its time in the kernel. On a 48 core system the execution time in the kernel jumps to 82%.

Talk:COMP 3000 Essay 2 2010 Question 1

2010-12-02T15:51:48Z

Kkashigi: /* Apache: Section 5.4 */

=Class and Notices=
(Nov. 30, 2010) Prof. Anil stated that we should focus on the 3 easiest to understand parts in section 5 and elaborate on them.

- Also, I, Daniel B., work Thursday night, so I will be finishing up as much of my part as I can for the essay before Thursday morning's class, then maybe we can all meet up in a lab in HP and put the finishing touches on the essay. I will be available online Wednesday night from about 6:30pm onwards and will be in the game dev lab or CCSS lounge Wednesday morning from about 11am to 2pm if anyone would like to meet up with me at those times.

- [[I suggest we meet up Thursday morning after Operating Systems in order to discuss and finalize the essay. Maybe we can even designate a lab for the group to meet up in. Any suggestions?]] - Daniel B.

- HP 3115 since there wont be a class in there (as its our tutorial and we know there won't be anyone there)
-- Go to Wireless Lab next to CCSS Lounge. Andrew and Dan B. will be there.

- If its all the same to you guys mind if I just join you via msn or iirc? Or phone if you really want. -Rannath

- I'm working today, but I'll be at a computer reading this page/contributing to my section. Depending on how busy I am, I should be able to get some significant writing in before 4pm today on my section and any additional sections required. RP

- I wont be there either. that does not mean i wont/cant contribute. I'll be on msn or you can just email me. -kirill

=Group members=
Patrick Young [mailto:Rannath@gmail.com Rannath@gmail.com]

Daniel Beimers [mailto:demongyro@gmail.com demongyro@gmail.com]

Andrew Bown [mailto:abown2@connect.carleton.ca abown2@connect.carleton.ca]

Kirill Kashigin [mailto:k.kashigin@gmail.com k.kashigin@gmail.com]

Rovic Perdon [mailto:rperdon@gmail.com rperdon@gmail.com]

Daniel Sont [mailto:dan.sont@gmail.com dan.sont@gmail.com]

=Methodology=
We should probably have our work verified by at least one group member before posting to the actual page

=To Do=
*Improve the grammar/structure of the paper section & add links to supplementary info
*Background Concepts -fill in info (fii)
*Contribution -fii
*Critique -fii
*References -fii

===Claim Sections===
* I claim Exim and memcached for background and critique -[[Rannath]]
* also per-core data structures, false sharing and unessesary locking for contribution -[[Rannath]]
* For starters I will take the Scalability Tutorial and gmake. Since the part for gmake is short in the paper, I will grab a few more sections later on. - [[Daniel B.]]
* Also, I will take sloppy counters as well - [[Daniel B.]]
* I'm gonna put some work into the apache and postgresql sections - kirill
* Just as a note Anil in class Thuesday the 30th of November said that we only need to explain 3 of the applications and not all 7 - [[Andrew]]
* I'll do the Research problem and contribution sections. - [[Andrew]]
* I will work on contribution - [[Rovic]]
* I'm gonna whip something up for 4.2 since there appears to be nothing mentioned about it. -kirill

* So here is the claims and unclaimed section. Add your name next to one if you want to take it on.
** gmake - Daniel B.
** memcached - Rannath
** Apache - Kirill
** [[(Exim, PostgreSQL, Metis, and Psearchy will not be needed as the professor said we only need to explain 3)]]
** Research Problem - Andrew
** Contribution - Rovic
** Critic, Style - Everyone
** Conclusion (also discussion) - Rannath, but I need someone to help flesh it out, I got the salient points down.
** References - Everyone

=Essay=
===Paper===
This paper was authored by - Silas Boyd-Wickizer, Austin T. Clements, Yandong Mao, Aleksey Pesterev, M. Frans Kaashoek, Robert Morris, and Nickolai Zeldovich.

They all work at MIT CSAIL.

[http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf The paper: An Analysis of Linux Scalability to Many Cores]

===Background Concepts===
Explain briefly the background concepts and ideas that your fellow classmates will need to know first in order to understand your assigned paper.

Ideas to explain:
- thread (maybe)
- Linux's move towards scalability precedes this paper. (assert this, no explanation needed, maybe a few examples)
- Summarize scalability tutorial (Section 4.1 of the paper) focus on what makes something (non-)scalable
- Describe the programs tested (what they do, how they're programmed (serial vs parallel), where to the do their processing)

====memchached: ''Section 3.2''====
memcached is an in-memory hash table server. One instance running on many cores is bottlenecked by an internal lock. The MIT team ran multiple instances to avoid the problem. Clients each connect to a single instance. This allows the server to simulate parallelism. With few requests, memcached spends 80% of its time in the kernel on one core, mostly processing packets.

====Apache: ''Section 3.3''====
Apache is a web server. In the case of this study, Apache has been configured to run a separate process on each core. Each process, in turn, has multiple threads (Making it a perfect example of parallel programming). One thread to service incoming connections and various other threads to service those connections. On a single core processor, Apache spends 60% of its execution time in the kernel.

====gmake: ''Section 3.5''====
gmake is an unofficial default benchmark in the Linux community which is used in this paper to build the Linux kernel. gmake is already quite parallel, creating more processes than cores, so that it can make proper use of multiple cores, and involves much reading and writing of files, as it is used to build the Linux kernel. gmake is limited in scalability due to the serial processes that run at the beginning and end of its execution. gmake spends much of its execution time with its compiler, but still spend 7.6% of its time in system time. [1]

===Research problem===
my references are just below because it is easier for numbering the data later.

As technological progress the number of core a main processor can have is increasing at an impressive rate. Soon personal computers will have so many cores that scalability will be an issue. There has to be a way that standard user level Linux kernel will scale with a 48-core system[1]. The problem with a standard Linux operating is they are not designed for massive scalability which will soon be a problem. The issue with scalability is that a solo core will perform much more work compared to a single core working with 47 other cores. Although traditional logic that situation makes sense because 48 cores are dividing the work. But when processing information a process the main goal is to finish so as long as possible every core should be doing a much work as possible.

To fix those scalability issues it is necessary to focus on three major areas: the Linux kernel, user level design and how application use of kernel services. The Linux kernel can be improved be to improve sharing and have the advantage of recent iterations are beginning to implement scalability features. On the user level design applications can be improved so that there is more focus on parallelism since some programs have not implements those improved features. The final aspect of improving scalability is how an application uses kernel services to share resources better so that different aspects of the program are not conflicting over the same services. All of the bottlenecks are found actually only take a little work to avoid.[1]

This research is based on much research which was created before in the development of scalability for UNIX system. The major developments from shared memory machines [2], wait-free synchronization to fast message passing have created a base set of techniques which can be used to improve scalability. These techniques have been incorporated in all major operation system including Linux, Mac OS X and Windows. Linux has been improved with kernel subsystems such as Read-Copy-Update which an algorithm for which is used to avoid locks and atomic instructions which lower scalability.[3] The is also an excellent base a research on Linux scalability studies to base this research paper. These paper include a on doing scalability on a 32-core machine. [4] That research can improve the results by learning from the experiments already performed by researchers. This research also aid identifying bottlenecks which speed up researching solutions for those bottlenecks.

[2] J. Kuskin, D. Ofelt, M. Heinrich, J. Heinlein, R. Simoni, K. Gharachorloo, J. Chapin, D. Nakahira, J. Baxter, M. Horowitz, A. Gupta, M. Rosenblum, and J. Hennessy. The Stanford FLASH multiprocessor. In Proc. of the 21st ISCA, pages 302–313,1994.

[3] P. E. McKenney, D. Sarma, A. Arcangeli, A. Kleen, O. Krieger, and R. Russell. Read-copy-update. In Proceedings of the Linux Symposium 2002, pages 338-367, Ottawa Ontario, June 2002

[4] C. Yan, Y. Chen, and S. Yuanchun. OSMark: A benchmark suite for understanding parallel scalability of operating systems on large scale multi-cores. In 2009 2nd International Conference on Computer Science and Information Technology, pages 313–317, 2009

==Section 4.1 problems:==
**The percentage of serialization in a program has a lot to do with how much an application can be sped up. As from the example in the paper, it seems to follow Amdahl's law (e.g. 25% serialization --> limit of 4x speedup).
**Types of serializing interactions found in the MOSBENCH apps:
***Locking of shared data structure - increasing # of cores --> increase in lock wait time
***Writing to shared memory - increasing # of cores --> increase in wait for cache coherence protocol
***Competing for space in shared hardware cache - increasing # of cores --> increase in cache miss rate
***Competing for shared hardware resources - increasing # of cores --> increase in wait for resources
***Not enough tasks for cores --> idle cores

===Contribution===
What was implemented? Why is it any better than what came before?

Summarize info from Section 4.2 onwards, maybe put graphs from Section 5 here to provide support for improvements (if that isn't unethical/illegal)?

- So long as we cite the paper and don't pretend the graphs are ours, we are ok, since we are writing an explanation/critic of the paper.

-I'm just using this as a notepad, do not copy/paste this section, I will put in a properly written set of paragraphs which will fit with the contribution questions asked. -RP

-==Work in Progress==-- -Rovic P.
This research contributes by evaluating the scalability discrepancies of applications programming and kernel programming. Key discoveries in this research show the effectiveness of the kernel in handling scaling amongst CPU cores. This has also shown that scaling in application programming should be more the focus. It has been shown that simple scaling techniques (list techniques) such as programming parallelism (look up more stuff to back this up and quotes). (Sloppy counter effectiveness, possible positive contributions, what has been used (internet search), what hasn’t been used.) Read conclusion, 2nd paragraph.

One reason the
required changes are modest is that stock Linux already
incorporates many modifications to improve scalability.
More speculatively, perhaps it is the case that Linux’s
system-call API is well suited to an implementation that
avoids unnecessary contention over kernel objects.

====Multicore packet processing: ''Section 4.2''====

====Sloppy counters: ''Section 4.3''====
Bottlenecks were encountered when the applications undergoing testing were referencing and updating shared counters for multiple cores. The solution in the paper is to use sloppy counters, which gets each core to track its own separate counts of references and uses a central shared counter to keep all counts on track. This is ideal because each core updates its counts by modifying its per-core counter, usually only needing access to its own local cache, cutting down on waiting for locks or serialization. Sloppy counters are also backwards-compatible with existing shared-counter code, making its implementation much easier to accomplish. The main disadvantages of the sloppy counters are that in situations where object de-allocation occurs often, because the de-allocation itself is an expensive operation, and the counters use up space proportional to the number of cores.

====Lock-free comparison: ''Section 4.4''====
This section describes a specific instance of unnecessary locking.

====Per-Core Data Structures: ''Section 4.5''====
Three centralized data structures were causing bottlenecks - a per-superblock list of open files, vfsmount table, the packet buffers free list. Each data structure was decentralized to per-core versions of itself. In the case of vfsmount the central data structure was maintained, and any per-core misses got written from the central table to the per-core table.

====Eliminating false sharing: ''Section 4.6''====
Misplaced variables on the cache cause different cores to request the same line to be read and written at the same time often enough to significantly impact performance. By moving the often written variable to another line the bottleneck was removed.

====Avoiding unnecessary locking: ''Section 4.7''====
Many locks/mutexes have special cases where they don't need to lock. Likewise mutexes can be split from locking the whole data structure to locking a part of it. Both these changes remove or reduce bottlenecks.

====Conclusion: ''Sections 6 & 7''====
Everything so far indicates that MOSBENCH application can scale to 48 cores. This scaling required a few modest changes to remove bottlenecks. The MIT team speculate that that trend will continue as the number of cores increase. They also state that things not bottlenecked by the CPU are harder to fix.

We can eliminate most kernel bottlenecks that the applications hits most often with minor changes. Most changes were well known methodology, with the exception of Sloppy counters. This study is limited by the removal of the IO bottleneck, but it does suggest that traditional implementations can be made scalable.

===Critique===
What is good and not-so-good about this paper? You may discuss both the style and content;
be sure to ground your discussion with specific references. Simple assertions that something is good or bad is not enough - you must explain why.

Since this is a "my implementation is better then your implementation" paper the "goodness" of content can be impartially determined by its fairness and the honesty of the authors.

====Content(Fairness): ''Section 5''====

=====memcached: ''Section 5.3''=====
memcached is treated with near perfect fairness in the paper. Its an in-memory service, so the ignored storage IO bottleneck does not affect it at all. Likewise the "stock" and "PK" implementations are given the same test suite, so there is no advantage given to either. memcached itself is non-scalable, so the MIT team was forced to run one instance per-core to keep up throughput. The FAQ at memcached.org's wiki suggests using multiple implementations per-server as a work around to another problem, which implies that running multiple instances of the server is the same, or nearly the same, as running one larger server [1]. In the end memcached was bottlenecked by the network card.

[1] memcached's wiki: http://code.google.com/p/memcached/wiki/FAQ#Can_I_use_different_size_caches_across_servers_and_will_memcache

=====Apache: ''Section 5.4''=====
Linux has a built in kernel flaw where network packets are forced to travel though multiple queues before they arrive at queue where they can be processed by the application. This imposes significant costs on multi-core systems due to queue locking costs. This flaw inherently diminishes the performance of Apache on multi-core system due to multiple threads spread across cores being forced to deal with these mutex (mutual exclusion) costs. For the sake of this experiment Apache had a separate instance on every core listening on different ports which is not a practical real world application but merely an attempt to implement better parallel execution on a traditional kernel. The patched kernel implementation of the network stack is also specific to the problem at hand, which is processing multiple short lived connections across multiple cores. Although this provides a performance increase in the given scenario, in more general applications network performance might suffer. These tests were also rigged to avoid bottlenecks in place by network and file storage hardware. Meaning, making the proposed modifications to the kernel wont necessarily produce the same increase in productivity as described in the article. This is very much evident in the test where performance degrades past 36 cores due to limitation of the networking hardware. ''Which is not a problem as the paper specifically states that there are hardware limitations.''

=====gmake: ''Section 5.6''=====
Since the inherent nature of gmake makes it quite parallel, the testing and updating that were attempted on gmake resulted in essentially the same scalability results for both the stock and modified kernel. The only change that was found was that gmake spent slightly less time at the system level because of the changes that were made to the system's caching. As stated in the paper, the execution time of gmake relies quite heavily on the compiler that is uses with gmake, so depending on which compiler was chosen, gmake could run worse or even slightly better. In any case, there seems to be no fairness concerns when it comes to the scalability testing of gmake as the same application load-out was used for all of the tests.

====Style====
Style Criterion (feel free to add I have no idea what should go here):
- does the paper present information out of order?
- does the paper present needless information?
- does the paper have any sections that are inherently confusing? Wrong? or use bad methodology?
- is the paper easy to read through, or does it change subjects repeatedly?
- does the paper have too many "long-winded" sentences, making it seem like the authors are just trying to add extra words to make it seem more important? - I think maybe limit this to run-on sentences.
- Check for grammar

===References===
[1] http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf

You will almost certainly have to refer to other resources; please cite these resources in the style of citation of the papers assigned (inlined numbered references). Place your bibliographic entries in this section.

gmake:

[http://www.gnu.org/software/make/manual/make.html gmake Manual]

[http://www.gnu.org/software/make/ gmake Main Page]

===Deprecated===
====Background Concepts====
* Exim: ''Section 3.1'':
**Exim is a mail server for Unix. It's fairly parallel. The server forks a new process for each connection and twice to deliver each message. It spends 69% of its time in the kernel on a single core.

* PostgreSQL: ''Section 3.4'':
**As implied by the name PostgreSQL is a SQL database. PostgreSQL starts a separate process for each connection and uses kernel locking interfaces extensively to provide concurrent access to the database. Due to bottlenecks introduced in its code and in the kernel code, the amount of time PostgreSQL spends in the kernel increases very rapidly with addition of new cores. On a single core system PostgreSQL spends only 1.5% of its time in the kernel. On a 48 core system the execution time in the kernel jumps to 82%.

* Psearchy: ''Section 3.6''

* Metis: ''Section 3.7''

Talk:COMP 3000 Essay 2 2010 Question 1

2010-12-02T15:26:52Z

Kkashigi: /* Claim Sections */

=Class and Notices=
(Nov. 30, 2010) Prof. Anil stated that we should focus on the 3 easiest to understand parts in section 5 and elaborate on them.

- Also, I, Daniel B., work Thursday night, so I will be finishing up as much of my part as I can for the essay before Thursday morning's class, then maybe we can all meet up in a lab in HP and put the finishing touches on the essay. I will be available online Wednesday night from about 6:30pm onwards and will be in the game dev lab or CCSS lounge Wednesday morning from about 11am to 2pm if anyone would like to meet up with me at those times.

- [[I suggest we meet up Thursday morning after Operating Systems in order to discuss and finalize the essay. Maybe we can even designate a lab for the group to meet up in. Any suggestions?]] - Daniel B.

- HP 3115 since there wont be a class in there (as its our tutorial and we know there won't be anyone there)
-- Go to Wireless Lab next to CCSS Lounge. Andrew and Dan B. will be there.

- If its all the same to you guys mind if I just join you via msn or iirc? Or phone if you really want. <- who is this?

- I'm working today, but I'll be at a computer reading this page/contributing to my section. Depending on how busy I am, I should be able to get some significant writing in before 4pm today on my section and any additional sections required. RP

- I wont be there either. that does not mean i wont/cant contribute. I'll be on msn or you can just email me. -kirill

=Group members=
Patrick Young [mailto:Rannath@gmail.com Rannath@gmail.com]

Daniel Beimers [mailto:demongyro@gmail.com demongyro@gmail.com]

Andrew Bown [mailto:abown2@connect.carleton.ca abown2@connect.carleton.ca]

Kirill Kashigin [mailto:k.kashigin@gmail.com k.kashigin@gmail.com]

Rovic Perdon [mailto:rperdon@gmail.com rperdon@gmail.com]

Daniel Sont [mailto:dan.sont@gmail.com dan.sont@gmail.com]

=Methodology=
We should probably have our work verified by at least one group member before posting to the actual page

=To Do=
*Improve the grammar/structure of the paper section & add links to supplementary info
*Background Concepts -fill in info (fii)
*Contribution -fii
*Critique -fii
*References -fii

===Claim Sections===
* I claim Exim and memcached for background and critique -[[Rannath]]
* also per-core data structures, false sharing and unessesary locking for contribution -[[Rannath]]
* For starters I will take the Scalability Tutorial and gmake. Since the part for gmake is short in the paper, I will grab a few more sections later on. - [[Daniel B.]]
* Also, I will take sloppy counters as well - [[Daniel B.]]
* I'm gonna put some work into the apache and postgresql sections - kirill
* Just as a note Anil in class Thuesday the 30th of November said that we only need to explain 3 of the applications and not all 7 - [[Andrew]]
* I'll do the Research problem and contribution sections. - [[Andrew]]
* I will work on contribution - [[Rovic]]
* I'm gonna whip something up for 4.2 since there appears to be nothing mentioned about it. -kirill

=Essay=
===Paper===
This paper was authored by - Silas Boyd-Wickizer, Austin T. Clements, Yandong Mao, Aleksey Pesterev, M. Frans Kaashoek, Robert Morris, and Nickolai Zeldovich.

They all work at MIT CSAIL.

[http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf The paper: An Analysis of Linux Scalability to Many Cores]

===Background Concepts===
Explain briefly the background concepts and ideas that your fellow classmates will need to know first in order to understand your assigned paper.

Ideas to explain:
- thread (maybe)
- Linux's move towards scalability precedes this paper. (assert this, no explanation needed, maybe a few examples)
- Summarize scalability tutorial (Section 4.1 of the paper) focus on what makes something (non-)scalable
- Describe the programs tested (what they do, how they're programmed (serial vs parallel), where to the do their processing)

====Exim: ''Section 3.1''====
Exim is a mail server for Unix. It's fairly parallel. The server forks a new process for each connection and twice to deliver each message. It spends 69% of its time in the kernel on a single core.

====memchached: ''Section 3.2''====
memcached is an in-memory hash table server. One instance running on many cores is bottlenecked by an internal lock. The MIT team ran multiple instances to avoid the problem. Clients each connect to a single instance. This allows the server to simulate parallelism. With few requests, memcached spends 80% of its time in the kernel on one core, mostly processing packets.

====Apache: ''Section 3.3''====
Apache is a web server. In the case of this study, Apache has been configured to run a separate process on each core. Each process, in turn, has multiple threads (Making it a perfect example of parallel programming). One thread to service incoming connections and various other threads to service those connections. On a single core processor, Apache spends 60% of its execution time in the kernel.

====PostgreSQL: ''Section 3.4''====
As implied by the name PostgreSQL is a SQL database. PostgreSQL starts a separate process for each connection and uses kernel locking interfaces extensively to provide concurrent access to the database. Due to bottlenecks introduced in its code and in the kernel code, the amount of time PostgreSQL spends in the kernel increases very rapidly with addition of new cores. On a single core system PostgreSQL spends only 1.5% of its time in the kernel. On a 48 core system the execution time in the kernel jumps to 82%.

====gmake: ''Section 3.5''====
gmake is an unofficial default benchmark in the Linux community which is used in this paper to build the Linux kernel. gmake is already quite parallel, creating more processes than cores, so that it can make proper use of multiple cores, and involves much reading and writing of files, as it is used to build the Linux kernel. gmake is limited in scalability due to the serial processes that run at the beginning and end of its execution. gmake spends much of its execution time with its compiler, but still spend 7.6% of its time in system time.

====Psearchy: ''Section 3.6''====

====Metis: ''Section 3.7''====

===Research problem===
my references are just below because it is easier for numbering the data later.

As technological progress the number of core a main processor can have is increasing at an impressive rate. Soon personal computers will have so many cores that scalability will be an issue. There has to be a way that standard user level Linux kernel will scale with a 48-core system[1]. The problem with a standard Linux operating is they are not designed for massive scalability which will soon be a problem. The issue with scalability is that a solo core will perform much more work compared to a single core working with 47 other cores. Although traditional logic that situation makes sense because 48 cores are dividing the work. But when processing information a process the main goal is to finish so as long as possible every core should be doing a much work as possible.

To fix those scalability issues it is necessary to focus on three major areas: the Linux kernel, user level design and how application use of kernel services. The Linux kernel can be improved be to improve sharing and have the advantage of recent iterations are beginning to implement scalability features. On the user level design applications can be improved so that there is more focus on parallelism since some programs have not implements those improved features. The final aspect of improving scalability is how an application uses kernel services to share resources better so that different aspects of the program are not conflicting over the same services. All of the bottlenecks are found actually only take a little work to avoid.[1]

This research is based on much research which was created before in the development of scalability for UNIX system. The major developments from shared memory machines [2], wait-free synchronization to fast message passing have created a base set of techniques which can be used to improve scalability. These techniques have been incorporated in all major operation system including Linux, Mac OS X and Windows. Linux has been improved with kernel subsystems such as Read-Copy-Update which an algorithm for which is used to avoid locks and atomic instructions which lower scalability.[3] The is also an excellent base a research on Linux scalability studies to base this research paper. These paper include a on doing scalability on a 32-core machine. [4] That research can improve the results by learning from the experiments already performed by researchers. This research also aid identifying bottlenecks which speed up researching solutions for those bottlenecks.

[2] J. Kuskin, D. Ofelt, M. Heinrich, J. Heinlein, R. Simoni, K. Gharachorloo, J. Chapin, D. Nakahira, J. Baxter, M. Horowitz, A. Gupta, M. Rosenblum, and J. Hennessy. The Stanford FLASH multiprocessor. In Proc. of the 21st ISCA, pages 302–313,1994.

[3] P. E. McKenney, D. Sarma, A. Arcangeli, A. Kleen, O. Krieger, and R. Russell. Read-copy-update. In Proceedings of the Linux Symposium 2002, pages 338-367, Ottawa Ontario, June 2002

[4] C. Yan, Y. Chen, and S. Yuanchun. OSMark: A benchmark suite for understanding parallel scalability of operating systems on large scale multi-cores. In 2009 2nd International Conference on Computer Science and Information Technology, pages 313–317, 2009

==Section 4.1 problems:==
**The percentage of serialization in a program has a lot to do with how much an application can be sped up. As from the example in the paper, it seems to follow Amdahl's law (e.g. 25% serialization --> limit of 4x speedup).
**Types of serializing interactions found in the MOSBENCH apps:
***Locking of shared data structure - increasing # of cores --> increase in lock wait time
***Writing to shared memory - increasing # of cores --> increase in wait for cache coherence protocol
***Competing for space in shared hardware cache - increasing # of cores --> increase in cache miss rate
***Competing for shared hardware resources - increasing # of cores --> increase in wait for resources
***Not enough tasks for cores --> idle cores

===Contribution===
What was implemented? Why is it any better than what came before?

Summarize info from Section 4.2 onwards, maybe put graphs from Section 5 here to provide support for improvements (if that isn't unethical/illegal)?

- So long as we cite the paper and don't pretend the graphs are ours, we are ok, since we are writing an explanation/critic of the paper.

-I'm just using this as a notepad, do not copy/paste this section, I will put in a properly written set of paragraphs which will fit with the contribution questions asked. -RP

-==Work in Progress==-- -Rovic P.
This research contributes by evaluating the scalability discrepancies of applications programming and kernel programming. Key discoveries in this research show the effectiveness of the kernel in handling scaling amongst CPU cores. This has also shown that scaling in application programming should be more the focus. It has been shown that simple scaling techniques (list techniques) such as programming parallelism (look up more stuff to back this up and quotes). (Sloppy counter effectiveness, possible positive contributions, what has been used (internet search), what hasn’t been used.) Read conclusion, 2nd paragraph.

One reason the
required changes are modest is that stock Linux already
incorporates many modifications to improve scalability.
More speculatively, perhaps it is the case that Linux’s
system-call API is well suited to an implementation that
avoids unnecessary contention over kernel objects.

====Multicore packet processing: ''Section 4.2''====

====Sloppy counters: ''Section 4.3''====
Bottlenecks were encountered when the applications undergoing testing were referencing and updating shared counters for multiple cores. The solution in the paper is to use sloppy counters, which gets each core to track its own separate counts of references and uses a central shared counter to keep all counts on track. This is ideal because each core updates its counts by modifying its per-core counter, usually only needing access to its own local cache, cutting down on waiting for locks or serialization. Sloppy counters are also backwards-compatible with existing shared-counter code, making its implementation much easier to accomplish. The main disadvantages of the sloppy counters are that in situations where object de-allocation occurs often, because the de-allocation itself is an expensive operation, and the counters use up space proportional to the number of cores.

====Lock-free comparison: ''Section 4.4''====
This section describes a specific instance of unnecessary locking.

====Per-Core Data Structures: ''Section 4.5''====
Three centralized data structures were causing bottlenecks - a per-superblock list of open files, vfsmount table, the packet buffers free list. Each data structure was decentralized to per-core versions of itself. In the case of vfsmount the central data structure was maintained, and any per-core misses got written from the central table to the per-core table.

====Eliminating false sharing: ''Section 4.6''====
Misplaced variables on the cache cause different cores to request the same line to be read and written at the same time often enough to significantly impact performance. By moving the often written variable to another line the bottleneck was removed.

====Avoiding unnecessary locking: ''Section 4.7''====
Many locks/mutexes have special cases where they don't need to lock. Likewise mutexes can be split from locking the whole data structure to locking a part of it. Both these changes remove or reduce bottlenecks.

====Conclusion====
Conclusion: we can make a traditional OS architecture scale better (at least to 48 cores) by removing bottlenecks, but hardware will still be a limiting factor to performance.

===Critique===
What is good and not-so-good about this paper? You may discuss both the style and content;
be sure to ground your discussion with specific references. Simple assertions that something is good or bad is not enough - you must explain why.

Since this is a "my implementation is better then your implementation" paper the "goodness" of content can be impartially determined by its fairness and the honesty of the authors.

====Content(Fairness): ''Section 5''====

=====memcached: ''Section 5.3''=====
memcached is treated with near perfect fairness in the paper. Its an in-memory service, so the ignored storage IO bottleneck does not affect it at all. Likewise the "stock" and "PK" implementations are given the same test suite, so there is no advantage given to either. memcached itself is non-scalable, so the MIT team was forced to run one instance per-core to keep up throughput. The FAQ at memcached.org's wiki suggests using multiple implementations per-server as a work around to another problem, which implies that running multiple instances of the server is the same, or nearly the same, as running one larger server [1]. In the end memcached was bottlenecked by the network card.

[1] memcached's wiki: http://code.google.com/p/memcached/wiki/FAQ#Can_I_use_different_size_caches_across_servers_and_will_memcache

=====Apache: ''Section 5.4''=====
Linux has a built in kernel flaw where network packets are forced to travel though multiple queues before they arrive at queue where they can be processed by the application. This imposes significant costs on multi-core systems due to queue locking costs. This flaw inherently diminishes the performance of Apache on multi-core system due to multiple threads spread across cores being forced to deal with these mutex (mutual exclusion) costs. For the sake of this experiment Apache had a separate instance on every core listening on different ports which is not a practical real world application but merely an attempt to implement better parallel execution on a traditional kernel. These tests were also rigged to avoid bottlenecks in place by network and file storage hardware. Meaning, making the proposed modifications to the kernel wont necessarily produce the same increase in productivity as described in the article. This is very much evident in the test where performance degrades past 36 cores due to limitation of the networking hardware. ''Which is not a problem as the paper specifically states that there are hardware limitations.''

=====gmake: ''Section 5.6''=====
Since the inherent nature of gmake makes it quite parallel, the testing and updating that were attempted on gmake resulted in essentially the same scalability results for both the stock and modified kernel. The only change that was found was that gmake spent slightly less time at the system level because of the changes that were made to the system's caching. As stated in the paper, the execution time of gmake relies quite heavily on the compiler that is uses with gmake, so depending on which compiler was chosen, gmake could run worse or even slightly better. In any case, there seems to be no fairness concerns when it comes to the scalability testing of gmake as the same application load-out was used for all of the tests.

====Style====
Style Criterion (feel free to add I have no idea what should go here):
- does the paper present information out of order?
- does the paper present needless information?
- does the paper have any sections that are inherently confusing? Wrong? or use bad methodology?
- is the paper easy to read through, or does it change subjects repeatedly?
- does the paper have too many "long-winded" sentences, making it seem like the authors are just trying to add extra words to make it seem more important? - I think maybe limit this to run-on sentences.
- Check for grammar

===References===
You will almost certainly have to refer to other resources; please cite these resources in the style of citation of the papers assigned (inlined numbered references). Place your bibliographic entries in this section.

gmake:

[http://www.gnu.org/software/make/manual/make.html gmake Manual]

[http://www.gnu.org/software/make/ gmake Main Page]

Talk:COMP 3000 Essay 2 2010 Question 1

2010-12-02T15:05:08Z

Kkashigi: /* Class and Notices */

=Class and Notices=
(Nov. 30, 2010) Prof. Anil stated that we should focus on the 3 easiest to understand parts in section 5 and elaborate on them.

- Also, I, Daniel B., work Thursday night, so I will be finishing up as much of my part as I can for the essay before Thursday morning's class, then maybe we can all meet up in a lab in HP and put the finishing touches on the essay. I will be available online Wednesday night from about 6:30pm onwards and will be in the game dev lab or CCSS lounge Wednesday morning from about 11am to 2pm if anyone would like to meet up with me at those times.

- [[I suggest we meet up Thursday morning after Operating Systems in order to discuss and finalize the essay. Maybe we can even designate a lab for the group to meet up in. Any suggestions?]] - Daniel B.

- HP 3115 since there wont be a class in there (as its our tutorial and we know there won't be anyone there)

- If its all the same to you guys mind if I just join you via msn or iirc? Or phone if you really want.

- I'm working today, but I'll be at a computer reading this page/contributing to my section. Depending on how busy I am, I should be able to get some significant writing in before 4pm today on my section and any additional sections required. RP

- I wont be there either. that does not mean i wont/cant contribute. I'll be on msn or you can just email me. -kirill

=Group members=
Patrick Young [mailto:Rannath@gmail.com Rannath@gmail.com]

Daniel Beimers [mailto:demongyro@gmail.com demongyro@gmail.com]

Andrew Bown [mailto:abown2@connect.carleton.ca abown2@connect.carleton.ca]

Kirill Kashigin [mailto:k.kashigin@gmail.com k.kashigin@gmail.com]

Rovic Perdon [mailto:rperdon@gmail.com rperdon@gmail.com]

Daniel Sont [mailto:dan.sont@gmail.com dan.sont@gmail.com]

=Methodology=
We should probably have our work verified by at least one group member before posting to the actual page

=To Do=
*Improve the grammar/structure of the paper section & add links to supplementary info
*Background Concepts -fill in info (fii)
*Contribution -fii
*Critique -fii
*References -fii

===Claim Sections===
* I claim Exim and memcached for background and critique -[[Rannath]]
* also per-core data structures, false sharing and unessesary locking for contribution -[[Rannath]]
* For starters I will take the Scalability Tutorial and gmake. Since the part for gmake is short in the paper, I will grab a few more sections later on. - [[Daniel B.]]
* Also, I will take sloppy counters as well - [[Daniel B.]]
* I'm gonna put some work into the apache and postgresql sections - kirill
* Just as a note Anil in class Thuesday the 30th of November said that we only need to explain 3 of the applications and not all 7 - [[Andrew]]
* I'll do the Research problem and contribution sections. - [[Andrew]]
* I will work on contribution - [[Rovic]]

=Essay=
===Paper===
This paper was authored by - Silas Boyd-Wickizer, Austin T. Clements, Yandong Mao, Aleksey Pesterev, M. Frans Kaashoek, Robert Morris, and Nickolai Zeldovich.

They all work at MIT CSAIL.

[http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf The paper: An Analysis of Linux Scalability to Many Cores]

===Background Concepts===
Explain briefly the background concepts and ideas that your fellow classmates will need to know first in order to understand your assigned paper.

Ideas to explain:
- thread (maybe)
- Linux's move towards scalability precedes this paper. (assert this, no explanation needed, maybe a few examples)
- Summarize scalability tutorial (Section 4.1 of the paper) focus on what makes something (non-)scalable
- Describe the programs tested (what they do, how they're programmed (serial vs parallel), where to the do their processing)

====Exim: ''Section 3.1''====
Exim is a mail server for Unix. It's fairly parallel. The server forks a new process for each connection and twice to deliver each message. It spends 69% of its time in the kernel on a single core.

====memchached: ''Section 3.2''====
memcached is an in-memory hash table server. One instance running on many cores is bottlenecked by an internal lock. The MIT team ran multiple instances to avoid the problem. Clients each connect to a single instance. This allows the server to simulate parallelism. With few requests, memcached spends 80% of its time in the kernel on one core, mostly processing packets.

====Apache: ''Section 3.3''====
Apache is a web server. In the case of this study, Apache has been configured to run a separate process on each core. Each process, in turn, has multiple threads (Making it a perfect example of parallel programming). One thread to service incoming connections and various other threads to service those connections. On a single core processor, Apache spends 60% of its execution time in the kernel.

====PostgreSQL: ''Section 3.4''====
As implied by the name PostgreSQL is a SQL database. PostgreSQL starts a separate process for each connection and uses kernel locking interfaces extensively to provide concurrent access to the database. Due to bottlenecks introduced in its code and in the kernel code, the amount of time PostgreSQL spends in the kernel increases very rapidly with addition of new cores. On a single core system PostgreSQL spends only 1.5% of its time in the kernel. On a 48 core system the execution time in the kernel jumps to 82%.

====gmake: ''Section 3.5''====
gmake is an unofficial default benchmark in the Linux community which is used in this paper to build the Linux kernel. gmake is already quite parallel, creating more processes than cores, so that it can make proper use of multiple cores, and involves much reading and writing of files, as it is used to build the Linux kernel. gmake is limited in scalability due to the serial processes that run at the beginning and end of its execution. gmake spends much of its execution time with its compiler, but still spend 7.6% of its time in system time.

====Psearchy: ''Section 3.6''====

====Metis: ''Section 3.7''====

===Research problem===
my references are just below because it is easier for numbering the data later.

As technological progress the number of core a main processor can have is increasing at an impressive rate. Soon personal computers will have so many cores that scalability will be an issue. There has to be a way that standard user level Linux kernel will scale with a 48-core system[1]. The problem with a standard Linux operating is they are not designed for massive scalability which will soon be a problem. The issue with scalability is that a solo core will perform much more work compared to a single core working with 47 other cores. Although traditional logic that situation makes sense because 48 cores are dividing the work. But when processing information a process the main goal is to finish so as long as possible every core should be doing a much work as possible.

To fix those scalability issues it is necessary to focus on three major areas: the Linux kernel, user level design and how application use of kernel services. The Linux kernel can be improved be to improve sharing and have the advantage of recent iterations are beginning to implement scalability features. On the user level design applications can be improved so that there is more focus on parallelism since some programs have not implements those improved features. The final aspect of improving scalability is how an application uses kernel services to share resources better so that different aspects of the program are not conflicting over the same services. All of the bottlenecks are found actually only take a little work to avoid.[1]

This research is based on much research which was created before in the development of scalability for UNIX system. The major developments from shared memory machines [2], wait-free synchronization to fast message passing have created a base set of techniques which can be used to improve scalability. These techniques have been incorporated in all major operation system including Linux, Mac OS X and Windows. Linux has been improved with kernel subsystems such as Read-Copy-Update which an algorithm for which is used to avoid locks and atomic instructions which lower scalability.[3] The is also an excellent base a research on Linux scalability studies to base this research paper. These paper include a on doing scalability on a 32-core machine. [4] That research can improve the results by learning from the experiments already performed by researchers. This research also aid identifying bottlenecks which speed up researching solutions for those bottlenecks.

[2] J. Kuskin, D. Ofelt, M. Heinrich, J. Heinlein, R. Simoni, K. Gharachorloo, J. Chapin, D. Nakahira, J. Baxter, M. Horowitz, A. Gupta, M. Rosenblum, and J. Hennessy. The Stanford FLASH multiprocessor. In Proc. of the 21st ISCA, pages 302–313,1994.

[3] P. E. McKenney, D. Sarma, A. Arcangeli, A. Kleen, O. Krieger, and R. Russell. Read-copy-update. In Proceedings of the Linux Symposium 2002, pages 338-367, Ottawa Ontario, June 2002

[4] C. Yan, Y. Chen, and S. Yuanchun. OSMark: A benchmark suite for understanding parallel scalability of operating systems on large scale multi-cores. In 2009 2nd International Conference on Computer Science and Information Technology, pages 313–317, 2009

==Section 4.1 problems:==
**The percentage of serialization in a program has a lot to do with how much an application can be sped up. As from the example in the paper, it seems to follow Amdahl's law (e.g. 25% serialization --> limit of 4x speedup).
**Types of serializing interactions found in the MOSBENCH apps:
***Locking of shared data structure - increasing # of cores --> increase in lock wait time
***Writing to shared memory - increasing # of cores --> increase in wait for cache coherence protocol
***Competing for space in shared hardware cache - increasing # of cores --> increase in cache miss rate
***Competing for shared hardware resources - increasing # of cores --> increase in wait for resources
***Not enough tasks for cores --> idle cores

===Contribution===
What was implemented? Why is it any better than what came before?

Summarize info from Section 4.2 onwards, maybe put graphs from Section 5 here to provide support for improvements (if that isn't unethical/illegal)?

- So long as we cite the paper and don't pretend the graphs are ours, we are ok, since we are writing an explanation/critic of the paper.

-I'm just using this as a notepad, do not copy/paste this section, I will put in a properly written set of paragraphs which will fit with the contribution questions asked. -RP

-==Work in Progress==-- -Rovic P.
This research contributes by evaluating the scalability discrepancies of applications programming and kernel programming. Key discoveries in this research show the effectiveness of the kernel in handling scaling amongst CPU cores. This has also shown that scaling in application programming should be more the focus. It has been shown that simple scaling techniques (list techniques) such as programming parallelism (look up more stuff to back this up and quotes). (Sloppy counter effectiveness, possible positive contributions, what has been used (internet search), what hasn’t been used.) Read conclusion, 2nd paragraph.

One reason the
required changes are modest is that stock Linux already
incorporates many modifications to improve scalability.
More speculatively, perhaps it is the case that Linux’s
system-call API is well suited to an implementation that
avoids unnecessary contention over kernel objects.

====Multicore packet processing: ''Section 4.2''====

====Sloppy counters: ''Section 4.3''====
Bottlenecks were encountered when the applications undergoing testing were referencing and updating shared counters for multiple cores. The solution in the paper is to use sloppy counters, which gets each core to track its own separate counts of references and uses a central shared counter to keep all counts on track. This is ideal because each core updates its counts by modifying its per-core counter, usually only needing access to its own local cache, cutting down on waiting for locks or serialization. Sloppy counters are also backwards-compatible with existing shared-counter code, making its implementation much easier to accomplish. The main disadvantages of the sloppy counters are that in situations where object de-allocation occurs often, because the de-allocation itself is an expensive operation, and the counters use up space proportional to the number of cores.

====Lock-free comparison: ''Section 4.4''====

====Per-Core Data Structures: ''Section 4.5''====
Three centralized data structures were causing bottlenecks - a per-superblock list of open files, vfsmount table, the packet buffers free list. Each data structure was decentralized to per-core versions of itself. In the case of vfsmount the central data structure was maintained, and any per-core misses got written from the central table to the per-core table.

====Eliminating false sharing: ''Section 4.6''====
Misplaced variables on the cache cause different cores to request the same line to be read and written at the same time often enough to significantly impact performance. By moving the often written variable to another line the bottleneck was removed.

====Avoiding unnecessary locking: ''Section 4.7''====
Many locks/mutexes have special cases where they don't need to lock. Likewise mutexes can be split from locking the whole data structure to locking a part of it. Both these changes remove or reduce bottlenecks.

====Conclusion====
Conclusion: we can make a traditional OS architecture scale better (at least to 48 cores) by removing bottlenecks, but hardware will still be a limiting factor to performance.

===Critique===
What is good and not-so-good about this paper? You may discuss both the style and content;
be sure to ground your discussion with specific references. Simple assertions that something is good or bad is not enough - you must explain why.

Since this is a "my implementation is better then your implementation" paper the "goodness" of content can be impartially determined by its fairness and the honesty of the authors.

====Content(Fairness): ''Section 5''====

=====memcached: ''Section 5.3''=====
memcached is treated with near perfect fairness in the paper. Its an in-memory service, so the ignored storage IO bottleneck does not affect it at all. Likewise the "stock" and "PK" implementations are given the same test suite, so there is no advantage given to either. memcached itself is non-scalable, so the MIT team was forced to run one instance per-core to keep up throughput. The FAQ at memcached.org's wiki suggests using multiple implementations per-server as a work around to another problem, which implies that running multiple instances of the server is the same, or nearly the same, as running one larger server [1]. In the end memcached was bottlenecked by a flaw in the network card.

[1] memcached's wiki: http://code.google.com/p/memcached/wiki/FAQ#Can_I_use_different_size_caches_across_servers_and_will_memcache

=====Apache: ''Section 5.4''=====
Linux has a built in kernel flaw where network packets are forced to travel though multiple queues before they arrive at queue where they can be processed by the application. This imposes significant costs on multi-core systems due to queue locking costs. This flaw inherently diminishes the performance of Apache on multi-core system due to multiple threads spread across cores being forced to deal with these mutex (mutual exclusion) costs. For the sake of this experiment Apache had a separate instance on every core listening on different ports which is not a practical real world application but merely an attempt to implement better parallel execution on a traditional kernel. These tests were also rigged to avoid bottlenecks in place by network and file storage hardware. Meaning, making the proposed modifications to the kernel wont necessarily produce the same increase in productivity as described in the article. This is very much evident in the test where performance degrades past 36 cores due to limitation of the networking hardware.

=====gmake: ''Section 5.6''=====
Since the inherent nature of gmake makes it quite parallel, the testing and updating that were attempted on gmake resulted in essentially the same scalability results for both the stock and modified kernel. The only change that was found was that gmake spent slightly less time at the system level because of the changes that were made to the system's caching. As stated in the paper, the execution time of gmake relies quite heavily on the compiler that is uses with gmake, so depending on which compiler was chosen, gmake could run worse or even slightly better. In any case, there seems to be no fairness concerns when it comes to the scalability testing of gmake as the same application load-out was used for all of the tests.

====Style====
Style Criterion (feel free to add I have no idea what should go here):
- does the paper present information out of order?
- does the paper present needless information?
- does the paper have any sections that are inherently confusing? Wrong? or use bad methodology?
- is the paper easy to read through, or does it change subjects repeatedly?
- does the paper have too many "long-winded" sentences, making it seem like the authors are just trying to add extra words to make it seem more important? - I think maybe limit this to run-on sentences.
- Check for grammar

===References===
You will almost certainly have to refer to other resources; please cite these resources in the style of citation of the papers assigned (inlined numbered references). Place your bibliographic entries in this section.

gmake:

[http://www.gnu.org/software/make/manual/make.html gmake Manual]

[http://www.gnu.org/software/make/ gmake Main Page]

Talk:COMP 3000 Essay 2 2010 Question 1

2010-12-02T00:03:13Z

Kkashigi: /* Apache: Section 5.4 */

Talk:COMP 3000 Essay 2 2010 Question 1

2010-12-02T00:01:18Z

Kkashigi: /* Apache: Section 5.4 */

Talk:COMP 3000 Essay 2 2010 Question 1

2010-12-01T23:57:01Z

Kkashigi: /* Apache: Section 3.3 */

Talk:COMP 3000 Essay 2 2010 Question 1

2010-12-01T22:27:52Z

Kkashigi:

Talk:COMP 3000 Essay 2 2010 Question 1

2010-11-30T21:11:58Z

Kkashigi: /* Claim Sections */

Talk:COMP 3000 Essay 2 2010 Question 1

2010-11-22T22:32:53Z

Kkashigi:

=Group members=
Patrick Young [mailto:Rannath@gmail.com Rannath@gmail.com]

Daniel Beimers [mailto:demongyro@gmail.com demongyro@gmail.com]

Andrew Bown [mailto:abown2@connect.carleton.ca abown2@connect.carleton.ca]

Kirill Kashigin [mailto:k.kashigin@gmail.com k.kashigin@gmail.com]

=Methodology=
We should probably have our work verified by at least one group member before posting to the actual page

=To Do=
*Improve the grammar/structure of the paper section
*Background Concepts -fill in info (fii)
*Research problem -fii
*Contribution -fii
*Critique -fii
*References -fii

===Claim Sections===
* I claim Exim and memcached for background and critique -[[Rannath]]
* also per-core data structures, false sharing and unessesary locking for contribution -[[Rannath]]

=Essay=
===Paper===
The paper's title, authors, and their affiliations. Include a link to the paper and any particularly helpful supplementary information.

Authors in order presented: Silas Boyd-Wickizer, Austin T. Clements, Yandong Mao, Aleksey Pesterev, M. Frans Kaashoek, Robert Morris, and Nickolai Zeldovich

affiliation: MIT CSAIL

[http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf An Analysis of Linux Scalability to Many Cores]

===Background Concepts===
Explain briefly the background concepts and ideas that your fellow classmates will need to know first in order to understand your assigned paper.

Ideas to explain:
#thread (maybe)
#Linux's move towards scalability precedes this paper. (assert this, no explanation needed, maybe a few examples)
#Summarize scalability tutorial (Section 4.1 of the paper)
#Describe the programs tested (what they do, how they're programmed (serial vs parallel), where to the do their processing)

=====Exim: ''Section 3.1''=====
Exim is a mail server for Unix. It's fairly parallel. The server forks a new process for each connection and twice to deliver each message. It spends 69% of its time in the kernel on a single core.

=====memchached: ''Section 3.2''=====
memcached is an in-memory hash table. memchached is very much not parallel, but can be made to be, just run multiple instances. Have clients worry about synchronizing data between the different instances. With few requests memcached does most of its processing at the network stack, 80% of its time on one core.

===Research problem===
What is the research problem being addressed by the paper? How does this problem relate to past related work?

Problem being addressed: scalability of current generation OS architecture, using Linux as an example. (?)

Summarize related works (Section 2, include links, expand information to have at least a summary of some related work)

===Contribution===
What was implemented? Why is it any better than what came before?

Summarize info from Section 4.2 onwards, maybe put graphs from Section 5 here to provide support for improvements (if that isn't unethical/illegal)?

Conclusion: we can make a traditional OS architecture scale (at least to 48 cores), we just have to remove bottlenecks.

=====Per-Core Data Structures=====
Three centralized data structures were causing bottlenecks - a per-superblock list of open files, vfsmount table, the packet buffers free list. Each data structure was decentralized to per-core versions of itself. In the case of vfsmount the central data structure was maintained, and any per-core misses got written from the central table to the per-core table.

=====Eliminating false sharing=====
Misplaced variables on the cache cause different cores to request the same line to be read and written at the same time often enough to significantly impact performance. By moving the often written variable to another line the bottleneck was removed.

=====Avoiding unnecessary locking=====
Many locks/mutexes have special cases where they don't need to lock. Likewise mutexes can be split from locking the whole data structure to locking a part of it. Both these changes remove or reduce bottlenecks.

===Critique===
What is good and not-so-good about this paper? You may discuss both the style and content; be sure to ground your discussion with specific references. Simple assertions that something is good or bad is not enough - you must explain why.

Since this is a "my implementation is better then your implementation" paper the "goodness" of content can be impartially determined by its fairness and the honesty of the authors.

Fairness criterion:
#does the test accurately describe real-world use-cases (or some set there-of)? (external fairness, maybe ignored for testing and benchmarking purposes, usually is too)
#does the test put all tested implementations through the same test? (internal fairness)

Style Criterion (feel free to add I have no idea what should go here):
#does the paper present information out of order?
#does the paper present needless information?
#does the paper have any sections that are inherently confusing?

=====Testing Method: ''Section 5''=====
Both the stock and new implementations use the same benchmarks, therefore internal fairness is preserved for all seven programs.

=====Exim: ''Section 5.2''=====
The test uses a relatively small number of connections, but that is also implicitly stated to be a non-issue - "as long as there are enough clients to keep Exim busy, the number of clients has little effect on performance."

This test is explicitly stated to be ignoring the real-world constraint of the IO bottleneck, thus is unfair when compared to real-world scenarios. The purpose was not to test the IO bottleneck. Therefore the unfairness to real-world scenarios is unimportant.

=====memcached: ''Section 5.3''=====
memcached has no explicit or implicit fairness concerns with respect to real-world scenarios.

===References===
You will almost certainly have to refer to other resources; please cite these resources in the style of citation of the papers assigned (inlined numbered references). Place your bibliographic entries in this section.

COMP 3000 Essay 1 2010 Question 10

2010-10-14T04:31:07Z

Kkashigi: /* Traditionally Optimized File Systems */

=Question=

How do the constraints of flash storage affect the design of flash-optimized file systems? Explain by contrasting with hard disk-based file systems.

=Answer=
''an introduction goes here''

==Flash Memory==
''basic info on flash memory''
===Constraints===
''--''

==Hard Disk drives==
''basic info on HDDs''
===Constraints===
''--''

==Traditionally Optimized File Systems==
''things traditional files systems do to optimize HDD read/write/etc''

Most conventional file systems are designed to me implemented on hard disk drives. This fact does not mean they cannot be implemented on a solid state drive (file storage that uses flash memory instead of magnetic discs). It would however, in many ways, defeat the purpose of using flash memory. The most consuming process for an HDD is seeking data by relocating the read-head and spinning the magnetic disk. A traditional file system optimizes the way it stores data by placing related blocks close-by on the disk to minimize mechanical movement within the HDD. One of the great advantages of flash memory, which accounts for its fast read speed, is that there is no need to seek data physically so there is no need to waste resources laying out the data in close proximity.
A traditional HDD file system will also attempt to defragment itself, moving blocks of data around for closer proximity on the magnetic disk. This process, although beneficial for HDD's, is harmful and inefficient for flash based storage. A flash optimal file system needs to reduce the amount of erase operations, since flash memory only has a limited amount of erase cycles as well as having very slow erase speeds.
When an HDD rewrites data to a physical location there is no need for it to erase the previously occupying data first, so a traditional disk based file system doesn't worry about erasing data from unused memory blocks. In contrast flash memory needs to first erase the data block before it can modify any of it contents. Since the erase procedure is extremely slow, its not practical to overwrite old data every time. It is also decremental to the life span of flash memory.
To maximize the potential of flash based memory the file system would have to write new data to empty memory blocks. This method would also call for some sort of garbage collection to erase unused blocks when the system is idle, which does not get implemented in conventional file systems since it is not needed.

==Flash Optimized File Systems==
''Flash Optimized files systems do to optimize HDD read/write/etc''

==Similarities==
''this could probably be titled better''

==Differences==
''ditto for this one''

=References=

=External links=

Hey guys,

This is what I've got so far... mostly based on wikipedia:

Flash memory has two limitations: it can only be erased in blocks and and it wears out after a certain number of erase cycles. Furthermore, a particular kind of Flash memory (NAND) is not able to provide random access.
As a result of these Flash based file-systems cannot be handled in the same way as disk-based file systems. Here are a few of the key differences:

- Because memory must be erased in blocks, its erasure tends to take up time. Consequently, it is necessary to time the erasures in a way so as not to interfere with the efficiency of the system’s other operations. This is is not a real concern with disk-based file-systems.
- A disk file-system needs to minimize the seeking time, but Flash file-system does not concern itself with this as it doesn’t have a disk.
- A flash system tries to distribute memory in such a way so as not to make a particular block of memory subject to a disproportionally large number of erasures. The purpose of this is to keep the block from wearing out prematurely. The result of it is that memory needs to be distributed differently than in a disk based file-system.
Log-sturctured file systems are thus best suited to dealing with flash memory (they apparently do all of the above things).

For the essay form, I'm thinking of doing a section about traditional hard-disk systems, another about flash-memory and a third about flash systems. At this point, I am imagining the thesis as something like, "Flash systems require a fundamentally different system architecture than disk-based systems due to their need to adapt to the constraints inherent in flash memory: specifically, due to that memory's limited life-span and block-based erasures." The argument would then talk about how these two differences directly lead to a new FS approach.

That's how I see it at the moment. Honestly, I don't like doing research about this kind of stuff, so my data isn't very deep. That said, if you guys could find more info and summarize it, I'm pretty sure that I could synthesize it all into a coherent essay.

Fedor

COMP 3000 Essay 1 2010 Question 10

2010-10-14T04:30:48Z

Kkashigi: /* Traditionally Optimized File Systems */

=Question=

How do the constraints of flash storage affect the design of flash-optimized file systems? Explain by contrasting with hard disk-based file systems.

=Answer=
''an introduction goes here''

==Flash Memory==
''basic info on flash memory''
===Constraints===
''--''

==Hard Disk drives==
''basic info on HDDs''
===Constraints===
''--''

==Traditionally Optimized File Systems==
''things traditional files systems do to optimize HDD read/write/etc''
<nowiki>
Most conventional file systems are designed to me implemented on hard disk drives. This fact does not mean they cannot be implemented on a solid state drive (file storage that uses flash memory instead of magnetic discs). It would however, in many ways, defeat the purpose of using flash memory. The most consuming process for an HDD is seeking data by relocating the read-head and spinning the magnetic disk. A traditional file system optimizes the way it stores data by placing related blocks close-by on the disk to minimize mechanical movement within the HDD. One of the great advantages of flash memory, which accounts for its fast read speed, is that there is no need to seek data physically so there is no need to waste resources laying out the data in close proximity.
A traditional HDD file system will also attempt to defragment itself, moving blocks of data around for closer proximity on the magnetic disk. This process, although beneficial for HDD's, is harmful and inefficient for flash based storage. A flash optimal file system needs to reduce the amount of erase operations, since flash memory only has a limited amount of erase cycles as well as having very slow erase speeds.
When an HDD rewrites data to a physical location there is no need for it to erase the previously occupying data first, so a traditional disk based file system doesn't worry about erasing data from unused memory blocks. In contrast flash memory needs to first erase the data block before it can modify any of it contents. Since the erase procedure is extremely slow, its not practical to overwrite old data every time. It is also decremental to the life span of flash memory.
To maximize the potential of flash based memory the file system would have to write new data to empty memory blocks. This method would also call for some sort of garbage collection to erase unused blocks when the system is idle, which does not get implemented in conventional file systems since it is not needed.
</nowiki>

==Flash Optimized File Systems==
''Flash Optimized files systems do to optimize HDD read/write/etc''

==Similarities==
''this could probably be titled better''

==Differences==
''ditto for this one''

=References=

=External links=

Hey guys,

This is what I've got so far... mostly based on wikipedia:

Flash memory has two limitations: it can only be erased in blocks and and it wears out after a certain number of erase cycles. Furthermore, a particular kind of Flash memory (NAND) is not able to provide random access.
As a result of these Flash based file-systems cannot be handled in the same way as disk-based file systems. Here are a few of the key differences:

- Because memory must be erased in blocks, its erasure tends to take up time. Consequently, it is necessary to time the erasures in a way so as not to interfere with the efficiency of the system’s other operations. This is is not a real concern with disk-based file-systems.
- A disk file-system needs to minimize the seeking time, but Flash file-system does not concern itself with this as it doesn’t have a disk.
- A flash system tries to distribute memory in such a way so as not to make a particular block of memory subject to a disproportionally large number of erasures. The purpose of this is to keep the block from wearing out prematurely. The result of it is that memory needs to be distributed differently than in a disk based file-system.
Log-sturctured file systems are thus best suited to dealing with flash memory (they apparently do all of the above things).

For the essay form, I'm thinking of doing a section about traditional hard-disk systems, another about flash-memory and a third about flash systems. At this point, I am imagining the thesis as something like, "Flash systems require a fundamentally different system architecture than disk-based systems due to their need to adapt to the constraints inherent in flash memory: specifically, due to that memory's limited life-span and block-based erasures." The argument would then talk about how these two differences directly lead to a new FS approach.

That's how I see it at the moment. Honestly, I don't like doing research about this kind of stuff, so my data isn't very deep. That said, if you guys could find more info and summarize it, I'm pretty sure that I could synthesize it all into a coherent essay.

Fedor

COMP 3000 Essay 1 2010 Question 10

2010-10-14T04:30:08Z

Kkashigi: /* Traditionally Optimized File Systems */

Talk:COMP 3000 Essay 1 2010 Question 10

2010-10-14T04:28:25Z

Kkashigi:

Hey all,

I think we should write down our emails here so we can further discuss stuff without having to login here.
('''***Note that discussions over email can't be counted towards your participation grade!***'''--[[User:Soma|Anil]])

Geoff Smith (gsmith0413@gmail.com) - gsmith6

Andrew Bujáki (abujaki [at] Connect or Live.ca)
***I'm usually on MSN(Live) for collaboration at nights, Just make sure to put in a little message about who you are when you're adding me. :)

I used Google Scholar and came to this page http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=812717&tag=1#
Which briefly touches on the issues of Flash memory. Specifically, inability to update in place, and limited write/erase cycles.

Inability to update in place could refer to the way the flash disk is programmed, instead of bit-by-bit, it is programmed block-by-block. A block would have to be erased and completely reprogrammed in order to flip one bit after it's been set.
http://en.wikipedia.org/wiki/Flash_memory#Block_erasure

Limited write/erase: Flash memory typically has a short lifespan if it's being used a lot. Writing and erasing the memory (Changing, updating, etc) Will wear it out. Flash memory has a finite amount of writes, (varying on manufacturer, models, etc), and once they've been used up, you'll get bad sectors, corrupt data, and generally be SOL.
http://en.wikipedia.org/wiki/Flash_memory#Memory_wear

Filesystems would have to be changed to play nicely with these constraints, where it must use blocks efficiently and nicely, and minimize writing/erasing as much as possible.

I found a paper that talks about the performance, capabilities and limitations of NAND flash storage.

Abstract: "This presentation provides an in-depth examination of the
fundamental theoretical performance, capabilities, and
limitations of NAND Flash-based Solid State Storage (SSS). The
tutorial will explore the raw performance capabilities of NAND
Flash, and limitations to performance imposed by mitigation of
reliability issues, interfaces, protocols, and technology types.
Best practices for system integration of SSS will be discussed.
Performance achievements will be reviewed for various
products and applications. "

Link: http://www.flashmemorysummit.com/English/Collaterals/Proceedings/2009/20090812_T1B_Smith.pdf

There's no Starting place like Wikipedia, even if you shouldn't source it.

http://en.wikipedia.org/wiki/Flash_Memory

http://en.wikipedia.org/wiki/LogFS

http://en.wikipedia.org/wiki/Hard_disk

http://en.wikipedia.org/wiki/Wear_leveling

http://en.wikipedia.org/wiki/Hot_spot_%28computer_science%29

http://en.wikipedia.org/wiki/Solid-state_drive

Hey Guys,

We really don't have much time to get this done. Lets meet tomorrow after class and get our bearings to do this properly.

Fedor

A few of us have Networking immediately after class. I know personally I won't be able to make anything set on Tuesday.
Additionally, he spoke briefly about hotspots on the disk for our question last week, where places on the disk would be written to far more often than others.
As well, for bibliographical citing, http://bibme.org is a wonderful resource for the popular formats (I.e. MLA). If it should come down to that.
~Andrew

===links===

Start Posting some stuff to source from:

http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1199079&tag=1
--"Introduction to flash memory"

http://portal.acm.org/citation.cfm?id=1244248
--"Wear Leveling" (it's about a proposed way of doing it, but explains a whole bunch of other things to do that)

http://portal.acm.org/citation.cfm?id=1731355
--"Online maintenance of very large random samples on flash storage" (ie dealing with the constraints of Flash Storage in a system that might actually be written to 100000 times)

http://vlsi.kaist.ac.kr/paper_list/2006_TC_CFFS.pdf
--"An Efficient NAND Flash File System for Flash Memory Storage" discuses shortcomings of using hard disk based file systems and current flash based file systems

http://maltiel-consulting.com/NAND_vs_NOR_Flash_Memory_Technology_Overview_Read_Write_Erase_speed_for_SLC_MLC_semiconductor_consulting_expert.pdf
--"NAND vs NOR Flash Memory" (note: i didn't get this off of Google scholar but it seems to be written by someone from Toshiba. is that ok?)

Hi everybody,

So here are the latest news. Geoff, Andrew and myself had a meeting after class today and came up with a plan for writing this thing.

We decided to have 3 parts:

1. What flash storage is, why its good but also why it must have the problems that it does (the assumption is that it must have them, why would it otherwise?)
[don't know much about this just now... basics include that there is NOR (reads slightly faster)and NAND (holds more, writes faster, erases much faster, lasts about ten times longer) flash with NAND being especially popular for storage (what's NOR good for?). Here, we'd ideally want to talk about why flash was invented (supposed as an alternative to slow ROM), why it was suitable for that, and how it works on a technical level. Then, we'd want to mention why this technical functionality was innovative and useful but also why it came with two serious set-backs: having a limited-number of re-write cycles and needing to erase a block at a time.]

Either way, Flash storage affords far faster fetch times than the traditional platter-based HDD, and stability of information in a sense. Where the data is not actually stored, but reprogrammed, in a sense, the data is more secure and is less likely to be erased easily. On that note, in order to flip a single bit, that entire block will need to be erased, then reprogrammed. In an 'old' HDD, let's say, When the HDD fails at the end of its life cycle, your data is gone. (unless you're willing to shell out $200/hr to have it recovered, yes I've seen companies in Ottawa that do this.) In a flash HDD, when it reaches the end of its life, it merely becomes read-only. Bugger for Databases, but useful for technical notes and archives, let's say.
With today's modern gaming computers, Flash memory can be good on quick load times, however with limited read-writes, it could afford better use to things that are not updated as frequently. I.e... Well I don't have a better example than a webserver hosting a company's CSS and scripts.
~Source: Years in the 'biz

Flash memory started out as a replacement for EPROMs. At the time EPROMs needed a UV photoemission to be erased while flash memory could be erased electronically. The first flash memory product came out in 1988 but it did not take off until the late 1990’s because it could not be reliable produced. NOR and NAND memory is named after the arrangement of the cells in the memory array. NOR based flash memory benefits from having very fast burst read times but slower write times. Due to the structure of NOR memory programs stored in NOR based memory can be executed without being loaded into RAM first. NAND flash memory has a very large storage capacity and can read and write large files relatively fast. NAND is more suited for storage while NOR memory is better suited for direct program execution such as in CMOS chips.
source: http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1199079&tag=1 , http://maltiel-consulting.com/NAND_vs_NOR_Flash_Memory_Technology_Overview_Read_Write_Erase_speed_for_SLC_MLC_semiconductor_consulting_expert.pdf

2. How a traditional disk-based file-system works and why the limitations of flash storage make the two a poor match
[the obvious answer seems to be that traditional file-systems could just write to whatever memory was available but if they did this with a flash file-systems, certain chunks of memory would become unusable before others and the memory would be more difficult to work with. Also, disk-based file systems need to deal with seeking times which means that they want to organize their data in such a way as to reduce those (by putting related things together?) - with Flash, this isn't really a problem and thus one constraint the less to be concerned with.]

3. How a log based file-system works and why this method of operation is so well suited to working with flash memory especially in light of the latter's inherent limitations
[...]

At this time, the plan is that Geoff will work on #3 today, Andrew will work on #1 tomorrow and I will work on #2 tomorrow. The three of us will make an effort to consult some somewhat more painfully technical literature in order to gain insight into our respective queries. Whatever insight we find will be posted here.

Then, we will meet again on Thursday after class to decide how to actually write the essay.

PS, if there is anybody in the group besides the three of us - let us know so you can find a way to contribute to this... as at least two of us are competent essayists, painfully technical research would on one or more of the above topics would be a great way to contribute... especially if you could post it here prior to one of us going over the same thing.

Fedor

-- I'm not that great (but absolutely horrid) at essays and I'm alright at research, but if nothing else I have Thursday off and nothing (else) that needs doing by Friday so I can probably spend a bunch of time working on it just before it's due. -- ''Nick L''

-- Hay sorry I was unable to attend the meeting after class today. I am not too good at writing essays as well but I am pretty good at summarizing and researching. I am not too sure at what you would like me to do. Right now I'll assume you need me to research/summarizing articles for the 3 topics above. If you need me to do anything else post it here. I'll be checking the discussion regularly until this due. once again sorry for missing the meeting-- Paul Cox.

-- Hey i'm also supposed to be in on this. Sorry i couldn't contribute sooner because i was playing catchup in my other classes. Let me know what i can do and i'll be on it asap. - kirill (k.kashigin@gmail.com)
update: i'm gonna be helping Fedor with #2

PS, this article http://docs.google.com/viewer?a=v&q=cache:E7-H_pv_18wJ:citeseerx.ist.psu.edu/viewdoc/download%3Fdoi%3D10.1.1.92.2279%26rep%3Drep1%26type%3Dpdf+flash+memory+and+disk-based+file+systems&hl=en&gl=ca&pid=bl&srcid=ADGEESgspy-jqIdLOpaLYlPPoM56kjLPwXcL3_eMbTTBRkI7PG0jQKl9vIieTAYHubPu0EdQ0V4ccaf_p0S_SnqKMirSIM0Qoq5E0NpLd0M7LAGaE51wkD0F55cRSkX8dnTqx_9Yx2E7&sig=AHIEtbS-yfGI9Y48DJ0WyEEhmsXInelRGw looks really useful for part 3.

PPS, and this article looks really great for understanding how log based file systems work: http://delivery.acm.org/10.1145/150000/146943/p26-rosenblum.pdf?key1=146943&key2=3656986821&coll=GUIDE&dl=GUIDE&CFID=108397378&CFTOKEN=72657973

Hey Luc (TA) here, Anandtech ran a series of articles on solid state drives that you guys might find useful. It mostly looked at hardware aspects but it gives some interesting insights on how to modify file systems to better support flash memory.

http://www.anandtech.com/cpuchipsets/intel/showdoc.aspx?i=3403

http://www.anandtech.com/storage/showdoc.aspx?i=3531&p=1

http://anandtech.com/storage/showdoc.aspx?i=3631

--[[User:3maisons|3maisons]] 19:44, 12 October 2010 (UTC)

Hey Paul&Kirill,

If one of you guys could help me out with #2, that would be really great. I was going to work on that tomorrow, but I also have another large assignment to deal with and not having to do this research would greatly ease my life. Moreover, I do intend to work on writing&polishing the essay on Thursday as I have a lot of experience with that and it far more than research. Let me know if either one of you can help me with this.

The other person could probably read over what Luc posted for us and see if it fits into our framework. Just be sure to state who is going to do what.

Nick,

Honestly, we really hope to have the research done by Thursday. If that is the only day that you are free and you're not a writer, I'm honestly not sure what you could do. Perhaps someone else can think of something.

- Fedor

I'm gonna have something for #2 up tonight. -kirill

So I found this article on Reddit, posted from Linux Weekly News on pretty much exactly what we are looking at. It's entitled "Solid-state storage devices and the block layer"

http://lwn.net/SubscriberLink/408428/68fa8465da45967a/ --[[User:Gsmith6|Gsmith6]] 20:36, 13 October 2010 (UTC)

I wasn't exactly sure how much information i was supposed to present but here's what i got for #2:

Most conventional file systems are designed to me implemented on hard disk drives. This fact does not mean they cannot be implemented on a solid state drive (file storage that uses flash memory instead of magnetic discs). It would however, in many ways, defeat the purpose of using flash memory. The most consuming process for an HDD is seeking data by relocating the read-head and spinning the magnetic disk. A traditional file system optimizes the way it stores data by placing related blocks close-by on the disk to minimize mechanical movement within the HDD. One of the great advantages of flash memory, which accounts for its fast read speed, is that there is no need to seek data physically so there is no need to waste resources laying out the data in close proximity.
A traditional HDD file system will also attempt to defragment itself, moving blocks of data around for closer proximity on the magnetic disk. This process, although beneficial for HDD's, is harmful and inefficient for flash based storage. A flash optimal file system needs to reduce the amount of erase operations, since flash memory only has a limited amount of erase cycles as well as having very slow erase speeds.
When an HDD rewrites data to a physical location there is no need for it to erase the previously occupying data first, so a traditional disk based file system doesn't worry about erasing data from unused memory blocks. In contrast flash memory needs to first erase the data block before it can modify any of it contents. Since the erase procedure is extremely slow, its not practical to overwrite old data every time. It is also decremental to the life span of flash memory.
To maximize the potential of flash based memory the file system would have to write new data to empty memory blocks. This method would also call for some sort of garbage collection to erase unused blocks when the system is idle, which does not get implemented in conventional file systems since it is not needed.

--kirill

Talk:COMP 3000 Essay 1 2010 Question 10

2010-10-13T15:25:06Z

Kkashigi:

Talk:COMP 3000 Essay 1 2010 Question 10

2010-10-13T02:33:05Z

Kkashigi:

Talk:COMP 3000 Essay 1 2010 Question 10

2010-10-12T22:28:03Z

Kkashigi: