Soma-notes - User contributions [en]

COMP 3000 Essay 2 2010 Question 1

2010-12-02T17:42:23Z

Abown: /* References */

=[http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf An Analysis of Linux Scalability to Many Cores]=
'''Authors:''' Silas Boyd-Wickizer, Austin T. Clements, Yandong Mao, Aleksey Pesterev, M. Frans Kaashoek, Robert Morris and Nickolai Zeldovich.

'''Affiliates:''' MIT CSAIL

=Background Concepts=
===memcached: ''Section 3.2''===
memcached is an in-memory hash table server. One instance of memcached running on many different cores is bottlenecked by an internal lock, which is avoided by the MIT team by running one instance per core. Clients each connect to a single instance of memcached, allowing the server to simulate parallelism without needing to make major changes to the application or kernel. With few requests, memcached spends 80% of its time in the kernel on one core, mostly processing packets.[[Foot1|1]]

===Apache: ''Section 3.3''===
Apache is a web server that has been used in previous Linux scalability studies. In the case of this study, Apache has been configured to run a separate process on each core. Each process, in turn, has multiple threads (making it a perfect example of parallel programming). Each process uses one of their threads to accepting incoming connections and others are used to process these connections. On a single core processor, Apache spends 60% of its execution time in the kernel.[[Foot1|1]]

===gmake: ''Section 3.5''===
gmake is an unofficial default benchmark in the Linux community which is used in this paper to build the Linux kernel. gmake takes a file called a makefile and processes its recipes for the requisite files to determine how and when to remake or recompile code. With a simple command -j or --jobs, gmake can process many of these recipes in parallel. Since gmake creates more processes than cores, it can make proper use of multiple cores to process the recipes.[2] Since gmake involves much reading and writing, in order to prevent bottlenecks due to the filesystem or hardware, the test cases use an in-memory filesystem tmpfs, which gives them a backdoor around the bottlenecks for testing purposes. In addition to this, gmake is limited in scalability due to the serial processes that run at the beginning and end of its execution, which limits its scalability to a small degree. gmake spends much of its execution time with its compiler, processing the recipes and recompiling code, but still spend 7.6% of its time in system time.[[Foot1|1]]

[2] http://www.gnu.org/software/make/manual/make.html

=Research problem=
What is the research problem being addressed by the paper? How does this problem relate to past related work?

=Contribution=
What are the research contribution(s) of this work? Specifically, what are the key research results, and what do they mean? (What was implemented? Why is it any better than what came before?)

=Critique=
What is good and not-so-good about this paper? You may discuss both the style and content; be sure to ground your discussion with specific references. Simple assertions that something is good or bad is not enough - you must explain why.

=References=
You will almost certainly have to refer to other resources; please cite these resources in the style of citation of the papers assigned (inlined numbered references). Place your bibliographic entries in this section.
:1Silas Boyd-Wickizer, Austin T. Clements, Yandong Mao, Aleksey Pesterev, M. Frans Kaashoek, Robert Morris, and Nickolai Zeldovich. An Analysis of Linux Scalability to Many Cores. MIT CSAIL,2010, http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf
:2J. Kuskin, D. Ofelt, M. Heinrich, J. Heinlein, R. Simoni, K. Gharachorloo, J. Chapin, D. Nakahira, J. Baxter, M. Horowitz, A. Gupta, M. Rosenblum, and J. Hennessy. The Stanford FLASH multiprocessor. In Proc. of the 21st ISCA, pages 302–313,1994.

:3P. E. McKenney, D. Sarma, A. Arcangeli, A. Kleen, O. Krieger, and R. Russell. Read-copy-update. In Proceedings of the Linux Symposium 2002, pages 338-367, Ottawa Ontario, June 2002

: 4C. Yan, Y. Chen, and S. Yuanchun. OSMark: A benchmark suite for understanding parallel scalability of operating systems on large scale multi-cores. In 2009 2nd International Conference on Computer Science and Information Technology, pages 313–317, 2009

COMP 3000 Essay 2 2010 Question 1

2010-12-02T17:42:00Z

Abown: /* References */

=[http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf An Analysis of Linux Scalability to Many Cores]=
'''Authors:''' Silas Boyd-Wickizer, Austin T. Clements, Yandong Mao, Aleksey Pesterev, M. Frans Kaashoek, Robert Morris and Nickolai Zeldovich.

'''Affiliates:''' MIT CSAIL

=Background Concepts=
===memcached: ''Section 3.2''===
memcached is an in-memory hash table server. One instance of memcached running on many different cores is bottlenecked by an internal lock, which is avoided by the MIT team by running one instance per core. Clients each connect to a single instance of memcached, allowing the server to simulate parallelism without needing to make major changes to the application or kernel. With few requests, memcached spends 80% of its time in the kernel on one core, mostly processing packets.[[Foot1|1]]

===Apache: ''Section 3.3''===
Apache is a web server that has been used in previous Linux scalability studies. In the case of this study, Apache has been configured to run a separate process on each core. Each process, in turn, has multiple threads (making it a perfect example of parallel programming). Each process uses one of their threads to accepting incoming connections and others are used to process these connections. On a single core processor, Apache spends 60% of its execution time in the kernel.[[Foot1|1]]

===gmake: ''Section 3.5''===
gmake is an unofficial default benchmark in the Linux community which is used in this paper to build the Linux kernel. gmake takes a file called a makefile and processes its recipes for the requisite files to determine how and when to remake or recompile code. With a simple command -j or --jobs, gmake can process many of these recipes in parallel. Since gmake creates more processes than cores, it can make proper use of multiple cores to process the recipes.[2] Since gmake involves much reading and writing, in order to prevent bottlenecks due to the filesystem or hardware, the test cases use an in-memory filesystem tmpfs, which gives them a backdoor around the bottlenecks for testing purposes. In addition to this, gmake is limited in scalability due to the serial processes that run at the beginning and end of its execution, which limits its scalability to a small degree. gmake spends much of its execution time with its compiler, processing the recipes and recompiling code, but still spend 7.6% of its time in system time.[[Foot1|1]]

[2] http://www.gnu.org/software/make/manual/make.html

=Research problem=
What is the research problem being addressed by the paper? How does this problem relate to past related work?

=Contribution=
What are the research contribution(s) of this work? Specifically, what are the key research results, and what do they mean? (What was implemented? Why is it any better than what came before?)

=Critique=
What is good and not-so-good about this paper? You may discuss both the style and content; be sure to ground your discussion with specific references. Simple assertions that something is good or bad is not enough - you must explain why.

=References=
You will almost certainly have to refer to other resources; please cite these resources in the style of citation of the papers assigned (inlined numbered references). Place your bibliographic entries in this section.
:1Silas Boyd-Wickizer, Austin T. Clements, Yandong Mao, Aleksey Pesterev, M. Frans Kaashoek, Robert Morris, and Nickolai Zeldovich. An Analysis of Linux Scalability to Many Cores. MIT CSAIL, http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf
:2J. Kuskin, D. Ofelt, M. Heinrich, J. Heinlein, R. Simoni, K. Gharachorloo, J. Chapin, D. Nakahira, J. Baxter, M. Horowitz, A. Gupta, M. Rosenblum, and J. Hennessy. The Stanford FLASH multiprocessor. In Proc. of the 21st ISCA, pages 302–313,1994.

:3P. E. McKenney, D. Sarma, A. Arcangeli, A. Kleen, O. Krieger, and R. Russell. Read-copy-update. In Proceedings of the Linux Symposium 2002, pages 338-367, Ottawa Ontario, June 2002

: 4C. Yan, Y. Chen, and S. Yuanchun. OSMark: A benchmark suite for understanding parallel scalability of operating systems on large scale multi-cores. In 2009 2nd International Conference on Computer Science and Information Technology, pages 313–317, 2009

COMP 3000 Essay 2 2010 Question 1

2010-12-02T17:34:04Z

Abown: /* References */

COMP 3000 Essay 2 2010 Question 1

2010-12-02T17:17:07Z

Abown: /* An Analysis of Linux Scalability to Many Cores */

COMP 3000 Essay 2 2010 Question 1

2010-12-02T17:11:47Z

Abown: /* An Analysis of Linux Scalability to Many Cores */

=An Analysis of Linux Scalability to Many Cores=
'''Authors:''' Silas Boyd-Wickizer, Austin T. Clements, Yandong Mao, Aleksey Pesterev, M. Frans Kaashoek, Robert Morris and Nickolai Zeldovich.

'''Affiliates:'''MIT CSAIL

'''Link to Paper:''' [http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf An Analysis of Linux Scalability to Many Cores]

=Background Concepts=
Explain briefly the background concepts and ideas that your fellow classmates will need to know first in order to understand your assigned paper.

=Research problem=
What is the research problem being addressed by the paper? How does this problem relate to past related work?

=Contribution=
What are the research contribution(s) of this work? Specifically, what are the key research results, and what do they mean? (What was implemented? Why is it any better than what came before?)

=Critique=
What is good and not-so-good about this paper? You may discuss both the style and content; be sure to ground your discussion with specific references. Simple assertions that something is good or bad is not enough - you must explain why.

=References=
You will almost certainly have to refer to other resources; please cite these resources in the style of citation of the papers assigned (inlined numbered references). Place your bibliographic entries in this section.

COMP 3000 Essay 2 2010 Question 1

2010-12-02T17:11:34Z

Abown: /* Paper */

=An Analysis of Linux Scalability to Many Cores=
'''Authors:''' Silas Boyd-Wickizer, Austin T. Clements, Yandong Mao, Aleksey Pesterev, M. Frans Kaashoek, Robert Morris and Nickolai Zeldovich.
'''Affiliates:'''MIT CSAIL
'''Link to Paper:''' [http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf An Analysis of Linux Scalability to Many Cores]

=Background Concepts=
Explain briefly the background concepts and ideas that your fellow classmates will need to know first in order to understand your assigned paper.

=Research problem=
What is the research problem being addressed by the paper? How does this problem relate to past related work?

=Contribution=
What are the research contribution(s) of this work? Specifically, what are the key research results, and what do they mean? (What was implemented? Why is it any better than what came before?)

=Critique=
What is good and not-so-good about this paper? You may discuss both the style and content; be sure to ground your discussion with specific references. Simple assertions that something is good or bad is not enough - you must explain why.

=References=
You will almost certainly have to refer to other resources; please cite these resources in the style of citation of the papers assigned (inlined numbered references). Place your bibliographic entries in this section.

Talk:COMP 3000 Essay 2 2010 Question 1

2010-12-02T15:55:37Z

Abown: /* Paper */

=Class and Notices=
(Nov. 30, 2010) Prof. Anil stated that we should focus on the 3 easiest to understand parts in section 5 and elaborate on them.

- Also, I, Daniel B., work Thursday night, so I will be finishing up as much of my part as I can for the essay before Thursday morning's class, then maybe we can all meet up in a lab in HP and put the finishing touches on the essay. I will be available online Wednesday night from about 6:30pm onwards and will be in the game dev lab or CCSS lounge Wednesday morning from about 11am to 2pm if anyone would like to meet up with me at those times.

- [[I suggest we meet up Thursday morning after Operating Systems in order to discuss and finalize the essay. Maybe we can even designate a lab for the group to meet up in. Any suggestions?]] - Daniel B.

- HP 3115 since there wont be a class in there (as its our tutorial and we know there won't be anyone there)
-- Go to Wireless Lab next to CCSS Lounge. Andrew and Dan B. will be there.

- If its all the same to you guys mind if I just join you via msn or iirc? Or phone if you really want. -Rannath

- I'm working today, but I'll be at a computer reading this page/contributing to my section. Depending on how busy I am, I should be able to get some significant writing in before 4pm today on my section and any additional sections required. RP

- I wont be there either. that does not mean i wont/cant contribute. I'll be on msn or you can just email me. -kirill

=Group members=
Patrick Young [mailto:Rannath@gmail.com Rannath@gmail.com]

Daniel Beimers [mailto:demongyro@gmail.com demongyro@gmail.com]

Andrew Bown [mailto:abown2@connect.carleton.ca abown2@connect.carleton.ca]

Kirill Kashigin [mailto:k.kashigin@gmail.com k.kashigin@gmail.com]

Rovic Perdon [mailto:rperdon@gmail.com rperdon@gmail.com]

Daniel Sont [mailto:dan.sont@gmail.com dan.sont@gmail.com]

=Methodology=
We should probably have our work verified by at least one group member before posting to the actual page

=To Do=
*Improve the grammar/structure of the paper section & add links to supplementary info
*Background Concepts -fill in info (fii)
*Contribution -fii
*Critique -fii
*References -fii

===Claim Sections===
* I claim Exim and memcached for background and critique -[[Rannath]]
* also per-core data structures, false sharing and unessesary locking for contribution -[[Rannath]]
* For starters I will take the Scalability Tutorial and gmake. Since the part for gmake is short in the paper, I will grab a few more sections later on. - [[Daniel B.]]
* Also, I will take sloppy counters as well - [[Daniel B.]]
* I'm gonna put some work into the apache and postgresql sections - kirill
* Just as a note Anil in class Thuesday the 30th of November said that we only need to explain 3 of the applications and not all 7 - [[Andrew]]
* I'll do the Research problem and contribution sections. - [[Andrew]]
* I will work on contribution - [[Rovic]]
* I'm gonna whip something up for 4.2 since there appears to be nothing mentioned about it. -kirill

* So here is the claims and unclaimed section. Add your name next to one if you want to take it on.
** gmake - Daniel B.
** memcached - Rannath
** Apache - Kirill
** [[(Exim, PostgreSQL, Metis, and Psearchy will not be needed as the professor said we only need to explain 3)]]
** Research Problem - Andrew
** Contribution - Rovic
** Critic, Style - Everyone
** Conclusion (also discussion) - Rannath, but I need someone to help flesh it out, I got the salient points down.
** References - Everyone
** Essay Conclusion - [[Nobody]]

=Essay=
==Paper==
This paper was authored by - Silas Boyd-Wickizer, Austin T. Clements, Yandong Mao, Aleksey Pesterev, M. Frans Kaashoek, Robert Morris, and Nickolai Zeldovich.

They all work at MIT CSAIL.

[http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf The paper: An Analysis of Linux Scalability to Many Cores]

==Background Concepts==
Explain briefly the background concepts and ideas that your fellow classmates will need to know first in order to understand your assigned paper.

Ideas to explain:
- thread (maybe)
- Linux's move towards scalability precedes this paper. (assert this, no explanation needed, maybe a few examples)
- Summarize scalability tutorial (Section 4.1 of the paper) focus on what makes something (non-)scalable
- Describe the programs tested (what they do, how they're programmed (serial vs parallel), where to the do their processing)

====memchached: ''Section 3.2''====
memcached is an in-memory hash table server. One instance running on many cores is bottlenecked by an internal lock. The MIT team ran one instance per-core to avoid the problem. Clients each connect to a single instance. This allows the server to simulate parallelism. With few requests, memcached spends 80% of its time in the kernel on one core, mostly processing packets.

====Apache: ''Section 3.3''====
Apache is a web server. In the case of this study, Apache has been configured to run a separate process on each core. Each process, in turn, has multiple threads (Making it a perfect example of parallel programming). One thread to service incoming connections and various other threads to service those connections. On a single core processor, Apache spends 60% of its execution time in the kernel.

====gmake: ''Section 3.5''====
gmake is an unofficial default benchmark in the Linux community which is used in this paper to build the Linux kernel. gmake is already quite parallel, creating more processes than cores, so that it can make proper use of multiple cores, and involves much reading and writing of files, as it is used to build the Linux kernel. gmake is limited in scalability due to the serial processes that run at the beginning and end of its execution. gmake spends much of its execution time with its compiler, but still spend 7.6% of its time in system time. [1]

==Research problem==
my references are just below because it is easier for numbering the data later.

As technological progress the number of core a main processor can have is increasing at an impressive rate. Soon personal computers will have so many cores that scalability will be an issue. There has to be a way that standard user level Linux kernel will scale with a 48-core system[1]. The problem with a standard Linux operating is they are not designed for massive scalability which will soon be a problem. The issue with scalability is that a solo core will perform much more work compared to a single core working with 47 other cores. Although traditional logic that situation makes sense because 48 cores are dividing the work. But when processing information a process the main goal is to finish so as long as possible every core should be doing a much work as possible.

To fix those scalability issues it is necessary to focus on three major areas: the Linux kernel, user level design and how application use of kernel services. The Linux kernel can be improved be to improve sharing and have the advantage of recent iterations are beginning to implement scalability features. On the user level design applications can be improved so that there is more focus on parallelism since some programs have not implements those improved features. The final aspect of improving scalability is how an application uses kernel services to share resources better so that different aspects of the program are not conflicting over the same services. All of the bottlenecks are found actually only take a little work to avoid.[1]

This research is based on much research which was created before in the development of scalability for UNIX system. The major developments from shared memory machines [2], wait-free synchronization to fast message passing have created a base set of techniques which can be used to improve scalability. These techniques have been incorporated in all major operation system including Linux, Mac OS X and Windows. Linux has been improved with kernel subsystems such as Read-Copy-Update which an algorithm for which is used to avoid locks and atomic instructions which lower scalability.[3] The is also an excellent base a research on Linux scalability studies to base this research paper. These paper include a on doing scalability on a 32-core machine. [4] That research can improve the results by learning from the experiments already performed by researchers. This research also aid identifying bottlenecks which speed up researching solutions for those bottlenecks.

[2] J. Kuskin, D. Ofelt, M. Heinrich, J. Heinlein, R. Simoni, K. Gharachorloo, J. Chapin, D. Nakahira, J. Baxter, M. Horowitz, A. Gupta, M. Rosenblum, and J. Hennessy. The Stanford FLASH multiprocessor. In Proc. of the 21st ISCA, pages 302–313,1994.

[3] P. E. McKenney, D. Sarma, A. Arcangeli, A. Kleen, O. Krieger, and R. Russell. Read-copy-update. In Proceedings of the Linux Symposium 2002, pages 338-367, Ottawa Ontario, June 2002

[4] C. Yan, Y. Chen, and S. Yuanchun. OSMark: A benchmark suite for understanding parallel scalability of operating systems on large scale multi-cores. In 2009 2nd International Conference on Computer Science and Information Technology, pages 313–317, 2009

==Contribution==
What was implemented? Why is it any better than what came before?

Summarize info from Section 4.2 onwards, maybe put graphs from Section 5 here to provide support for improvements (if that isn't unethical/illegal)?

- So long as we cite the paper and don't pretend the graphs are ours, we are ok, since we are writing an explanation/critic of the paper.

-I'm just using this as a notepad, do not copy/paste this section, I will put in a properly written set of paragraphs which will fit with the contribution questions asked. -RP

-==Work in Progress==-- -Rovic P.
This research contributes by evaluating the scalability discrepancies of applications programming and kernel programming. Key discoveries in this research show the effectiveness of the kernel in handling scaling amongst CPU cores. This has also shown that scaling in application programming should be more the focus. It has been shown that simple scaling techniques (list techniques) such as programming parallelism (look up more stuff to back this up and quotes). (Sloppy counter effectiveness, possible positive contributions, what has been used (internet search), what hasn’t been used.) Read conclusion, 2nd paragraph.

One reason the
required changes are modest is that stock Linux already
incorporates many modifications to improve scalability.
More speculatively, perhaps it is the case that Linux’s
system-call API is well suited to an implementation that
avoids unnecessary contention over kernel objects.

===Section 4.1 problems:===
**The percentage of serialization in a program has a lot to do with how much an application can be sped up. As from the example in the paper, it seems to follow Amdahl's law (e.g. 25% serialization --> limit of 4x speedup).
**Types of serializing interactions found in the MOSBENCH apps:
***Locking of shared data structure - increasing # of cores --> increase in lock wait time
***Writing to shared memory - increasing # of cores --> increase in wait for cache coherence protocol
***Competing for space in shared hardware cache - increasing # of cores --> increase in cache miss rate
***Competing for shared hardware resources - increasing # of cores --> increase in wait for resources
***Not enough tasks for cores --> idle cores

====Multicore packet processing: ''Section 4.2''====

====Sloppy counters: ''Section 4.3''====
Bottlenecks were encountered when the applications undergoing testing were referencing and updating shared counters for multiple cores. The solution in the paper is to use sloppy counters, which gets each core to track its own separate counts of references and uses a central shared counter to keep all counts on track. This is ideal because each core updates its counts by modifying its per-core counter, usually only needing access to its own local cache, cutting down on waiting for locks or serialization. Sloppy counters are also backwards-compatible with existing shared-counter code, making its implementation much easier to accomplish. The main disadvantages of the sloppy counters are that in situations where object de-allocation occurs often, because the de-allocation itself is an expensive operation, and the counters use up space proportional to the number of cores.

====Lock-free comparison: ''Section 4.4''====
This section describes a specific instance of unnecessary locking.

====Per-Core Data Structures: ''Section 4.5''====
Three centralized data structures were causing bottlenecks - a per-superblock list of open files, vfsmount table, the packet buffers free list. Each data structure was decentralized to per-core versions of itself. In the case of vfsmount the central data structure was maintained, and any per-core misses got written from the central table to the per-core table.

====Eliminating false sharing: ''Section 4.6''====
Misplaced variables on the cache cause different cores to request the same line to be read and written at the same time often enough to significantly impact performance. By moving the often written variable to another line the bottleneck was removed.

====Avoiding unnecessary locking: ''Section 4.7''====
Many locks/mutexes have special cases where they don't need to lock. Likewise mutexes can be split from locking the whole data structure to locking a part of it. Both these changes remove or reduce bottlenecks.

====Conclusion: ''Sections 6 & 7''====
Everything so far indicates that MOSBENCH application can scale to 48 cores. This scaling required a few modest changes to remove bottlenecks. The MIT team speculate that that trend will continue as the number of cores increase. They also state that things not bottlenecked by the CPU are harder to fix.

We can eliminate most kernel bottlenecks that the applications hits most often with minor changes. Most changes were well known methodology, with the exception of Sloppy counters. This study is limited by the removal of the IO bottleneck, but it does suggest that traditional implementations can be made scalable.

==Critique==
What is good and not-so-good about this paper? You may discuss both the style and content;
be sure to ground your discussion with specific references. Simple assertions that something is good or bad is not enough - you must explain why.

Since this is a "my implementation is better then your implementation" paper the "goodness" of content can be impartially determined by its fairness and the honesty of the authors.

====Content(Fairness): ''Section 5''====

=====memcached: ''Section 5.3''=====
memcached is treated with near perfect fairness in the paper. Its an in-memory service, so the ignored storage IO bottleneck does not affect it at all. Likewise the "stock" and "PK" implementations are given the same test suite, so there is no advantage given to either. memcached itself is non-scalable, so the MIT team was forced to run one instance per-core to keep up throughput. The FAQ at memcached.org's wiki suggests using multiple implementations per-server as a work around to another problem, which implies that running multiple instances of the server is the same, or nearly the same, as running one larger server [1]. In the end memcached was bottlenecked by the network card.

[1] memcached's wiki: http://code.google.com/p/memcached/wiki/FAQ#Can_I_use_different_size_caches_across_servers_and_will_memcache

=====Apache: ''Section 5.4''=====
Linux has a built in kernel flaw where network packets are forced to travel though multiple queues before they arrive at queue where they can be processed by the application. This imposes significant costs on multi-core systems due to queue locking costs. This flaw inherently diminishes the performance of Apache on multi-core system due to multiple threads spread across cores being forced to deal with these mutex (mutual exclusion) costs. For the sake of this experiment Apache had a separate instance on every core listening on different ports which is not a practical real world application but merely an attempt to implement better parallel execution on a traditional kernel. The patched kernel implementation of the network stack is also specific to the problem at hand, which is processing multiple short lived connections across multiple cores. Although this provides a performance increase in the given scenario, in more general applications network performance might suffer. These tests were also rigged to avoid bottlenecks in place by network and file storage hardware. Meaning, making the proposed modifications to the kernel wont necessarily produce the same increase in productivity as described in the article. This is very much evident in the test where performance degrades past 36 cores due to limitation of the networking hardware. ''Which is not a problem as the paper specifically states that there are hardware limitations.''

=====gmake: ''Section 5.6''=====
Since the inherent nature of gmake makes it quite parallel, the testing and updating that were attempted on gmake resulted in essentially the same scalability results for both the stock and modified kernel. The only change that was found was that gmake spent slightly less time at the system level because of the changes that were made to the system's caching. As stated in the paper, the execution time of gmake relies quite heavily on the compiler that is uses with gmake, so depending on which compiler was chosen, gmake could run worse or even slightly better. In any case, there seems to be no fairness concerns when it comes to the scalability testing of gmake as the same application load-out was used for all of the tests.

====Style====
Style Criterion (feel free to add I have no idea what should go here):
- does the paper present information out of order?
- does the paper present needless information?
- does the paper have any sections that are inherently confusing? Wrong? or use bad methodology?
- is the paper easy to read through, or does it change subjects repeatedly?
- does the paper have too many "long-winded" sentences, making it seem like the authors are just trying to add extra words to make it seem more important? - I think maybe limit this to run-on sentences.
- Check for grammar

===Conclusion===
[[Of the entire essay...]]

===References===
[1] Silas Boyd-Wickizer et al. "An Analysis of Linux Scalability to Many Cores". In ''OSDI '10, 9th USENIX Symposium on OS Design and Implementation'', Vancouver, BC, Canada, 2010. http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf.

You will almost certainly have to refer to other resources; please cite these resources in the style of citation of the papers assigned (inlined numbered references). Place your bibliographic entries in this section.

gmake:

[http://www.gnu.org/software/make/manual/make.html gmake Manual]

[http://www.gnu.org/software/make/ gmake Main Page]

===Deprecated===
====Background Concepts====
* Exim: ''Section 3.1'':
**Exim is a mail server for Unix. It's fairly parallel. The server forks a new process for each connection and twice to deliver each message. It spends 69% of its time in the kernel on a single core.

* PostgreSQL: ''Section 3.4'':
**As implied by the name PostgreSQL is a SQL database. PostgreSQL starts a separate process for each connection and uses kernel locking interfaces extensively to provide concurrent access to the database. Due to bottlenecks introduced in its code and in the kernel code, the amount of time PostgreSQL spends in the kernel increases very rapidly with addition of new cores. On a single core system PostgreSQL spends only 1.5% of its time in the kernel. On a 48 core system the execution time in the kernel jumps to 82%.

Talk:COMP 3000 Essay 2 2010 Question 1

2010-12-02T15:55:24Z

Abown: /* Background Concepts */

=Class and Notices=
(Nov. 30, 2010) Prof. Anil stated that we should focus on the 3 easiest to understand parts in section 5 and elaborate on them.

- Also, I, Daniel B., work Thursday night, so I will be finishing up as much of my part as I can for the essay before Thursday morning's class, then maybe we can all meet up in a lab in HP and put the finishing touches on the essay. I will be available online Wednesday night from about 6:30pm onwards and will be in the game dev lab or CCSS lounge Wednesday morning from about 11am to 2pm if anyone would like to meet up with me at those times.

- [[I suggest we meet up Thursday morning after Operating Systems in order to discuss and finalize the essay. Maybe we can even designate a lab for the group to meet up in. Any suggestions?]] - Daniel B.

- HP 3115 since there wont be a class in there (as its our tutorial and we know there won't be anyone there)
-- Go to Wireless Lab next to CCSS Lounge. Andrew and Dan B. will be there.

- If its all the same to you guys mind if I just join you via msn or iirc? Or phone if you really want. -Rannath

- I'm working today, but I'll be at a computer reading this page/contributing to my section. Depending on how busy I am, I should be able to get some significant writing in before 4pm today on my section and any additional sections required. RP

- I wont be there either. that does not mean i wont/cant contribute. I'll be on msn or you can just email me. -kirill

=Group members=
Patrick Young [mailto:Rannath@gmail.com Rannath@gmail.com]

Daniel Beimers [mailto:demongyro@gmail.com demongyro@gmail.com]

Andrew Bown [mailto:abown2@connect.carleton.ca abown2@connect.carleton.ca]

Kirill Kashigin [mailto:k.kashigin@gmail.com k.kashigin@gmail.com]

Rovic Perdon [mailto:rperdon@gmail.com rperdon@gmail.com]

Daniel Sont [mailto:dan.sont@gmail.com dan.sont@gmail.com]

=Methodology=
We should probably have our work verified by at least one group member before posting to the actual page

=To Do=
*Improve the grammar/structure of the paper section & add links to supplementary info
*Background Concepts -fill in info (fii)
*Contribution -fii
*Critique -fii
*References -fii

===Claim Sections===
* I claim Exim and memcached for background and critique -[[Rannath]]
* also per-core data structures, false sharing and unessesary locking for contribution -[[Rannath]]
* For starters I will take the Scalability Tutorial and gmake. Since the part for gmake is short in the paper, I will grab a few more sections later on. - [[Daniel B.]]
* Also, I will take sloppy counters as well - [[Daniel B.]]
* I'm gonna put some work into the apache and postgresql sections - kirill
* Just as a note Anil in class Thuesday the 30th of November said that we only need to explain 3 of the applications and not all 7 - [[Andrew]]
* I'll do the Research problem and contribution sections. - [[Andrew]]
* I will work on contribution - [[Rovic]]
* I'm gonna whip something up for 4.2 since there appears to be nothing mentioned about it. -kirill

* So here is the claims and unclaimed section. Add your name next to one if you want to take it on.
** gmake - Daniel B.
** memcached - Rannath
** Apache - Kirill
** [[(Exim, PostgreSQL, Metis, and Psearchy will not be needed as the professor said we only need to explain 3)]]
** Research Problem - Andrew
** Contribution - Rovic
** Critic, Style - Everyone
** Conclusion (also discussion) - Rannath, but I need someone to help flesh it out, I got the salient points down.
** References - Everyone
** Essay Conclusion - [[Nobody]]

=Essay=
===Paper===
This paper was authored by - Silas Boyd-Wickizer, Austin T. Clements, Yandong Mao, Aleksey Pesterev, M. Frans Kaashoek, Robert Morris, and Nickolai Zeldovich.

They all work at MIT CSAIL.

[http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf The paper: An Analysis of Linux Scalability to Many Cores]

==Background Concepts==
Explain briefly the background concepts and ideas that your fellow classmates will need to know first in order to understand your assigned paper.

Ideas to explain:
- thread (maybe)
- Linux's move towards scalability precedes this paper. (assert this, no explanation needed, maybe a few examples)
- Summarize scalability tutorial (Section 4.1 of the paper) focus on what makes something (non-)scalable
- Describe the programs tested (what they do, how they're programmed (serial vs parallel), where to the do their processing)

====memchached: ''Section 3.2''====
memcached is an in-memory hash table server. One instance running on many cores is bottlenecked by an internal lock. The MIT team ran one instance per-core to avoid the problem. Clients each connect to a single instance. This allows the server to simulate parallelism. With few requests, memcached spends 80% of its time in the kernel on one core, mostly processing packets.

====Apache: ''Section 3.3''====
Apache is a web server. In the case of this study, Apache has been configured to run a separate process on each core. Each process, in turn, has multiple threads (Making it a perfect example of parallel programming). One thread to service incoming connections and various other threads to service those connections. On a single core processor, Apache spends 60% of its execution time in the kernel.

====gmake: ''Section 3.5''====
gmake is an unofficial default benchmark in the Linux community which is used in this paper to build the Linux kernel. gmake is already quite parallel, creating more processes than cores, so that it can make proper use of multiple cores, and involves much reading and writing of files, as it is used to build the Linux kernel. gmake is limited in scalability due to the serial processes that run at the beginning and end of its execution. gmake spends much of its execution time with its compiler, but still spend 7.6% of its time in system time. [1]

==Research problem==
my references are just below because it is easier for numbering the data later.

As technological progress the number of core a main processor can have is increasing at an impressive rate. Soon personal computers will have so many cores that scalability will be an issue. There has to be a way that standard user level Linux kernel will scale with a 48-core system[1]. The problem with a standard Linux operating is they are not designed for massive scalability which will soon be a problem. The issue with scalability is that a solo core will perform much more work compared to a single core working with 47 other cores. Although traditional logic that situation makes sense because 48 cores are dividing the work. But when processing information a process the main goal is to finish so as long as possible every core should be doing a much work as possible.

To fix those scalability issues it is necessary to focus on three major areas: the Linux kernel, user level design and how application use of kernel services. The Linux kernel can be improved be to improve sharing and have the advantage of recent iterations are beginning to implement scalability features. On the user level design applications can be improved so that there is more focus on parallelism since some programs have not implements those improved features. The final aspect of improving scalability is how an application uses kernel services to share resources better so that different aspects of the program are not conflicting over the same services. All of the bottlenecks are found actually only take a little work to avoid.[1]

This research is based on much research which was created before in the development of scalability for UNIX system. The major developments from shared memory machines [2], wait-free synchronization to fast message passing have created a base set of techniques which can be used to improve scalability. These techniques have been incorporated in all major operation system including Linux, Mac OS X and Windows. Linux has been improved with kernel subsystems such as Read-Copy-Update which an algorithm for which is used to avoid locks and atomic instructions which lower scalability.[3] The is also an excellent base a research on Linux scalability studies to base this research paper. These paper include a on doing scalability on a 32-core machine. [4] That research can improve the results by learning from the experiments already performed by researchers. This research also aid identifying bottlenecks which speed up researching solutions for those bottlenecks.

[2] J. Kuskin, D. Ofelt, M. Heinrich, J. Heinlein, R. Simoni, K. Gharachorloo, J. Chapin, D. Nakahira, J. Baxter, M. Horowitz, A. Gupta, M. Rosenblum, and J. Hennessy. The Stanford FLASH multiprocessor. In Proc. of the 21st ISCA, pages 302–313,1994.

[3] P. E. McKenney, D. Sarma, A. Arcangeli, A. Kleen, O. Krieger, and R. Russell. Read-copy-update. In Proceedings of the Linux Symposium 2002, pages 338-367, Ottawa Ontario, June 2002

[4] C. Yan, Y. Chen, and S. Yuanchun. OSMark: A benchmark suite for understanding parallel scalability of operating systems on large scale multi-cores. In 2009 2nd International Conference on Computer Science and Information Technology, pages 313–317, 2009

==Contribution==
What was implemented? Why is it any better than what came before?

Summarize info from Section 4.2 onwards, maybe put graphs from Section 5 here to provide support for improvements (if that isn't unethical/illegal)?

- So long as we cite the paper and don't pretend the graphs are ours, we are ok, since we are writing an explanation/critic of the paper.

-I'm just using this as a notepad, do not copy/paste this section, I will put in a properly written set of paragraphs which will fit with the contribution questions asked. -RP

-==Work in Progress==-- -Rovic P.
This research contributes by evaluating the scalability discrepancies of applications programming and kernel programming. Key discoveries in this research show the effectiveness of the kernel in handling scaling amongst CPU cores. This has also shown that scaling in application programming should be more the focus. It has been shown that simple scaling techniques (list techniques) such as programming parallelism (look up more stuff to back this up and quotes). (Sloppy counter effectiveness, possible positive contributions, what has been used (internet search), what hasn’t been used.) Read conclusion, 2nd paragraph.

One reason the
required changes are modest is that stock Linux already
incorporates many modifications to improve scalability.
More speculatively, perhaps it is the case that Linux’s
system-call API is well suited to an implementation that
avoids unnecessary contention over kernel objects.

===Section 4.1 problems:===
**The percentage of serialization in a program has a lot to do with how much an application can be sped up. As from the example in the paper, it seems to follow Amdahl's law (e.g. 25% serialization --> limit of 4x speedup).
**Types of serializing interactions found in the MOSBENCH apps:
***Locking of shared data structure - increasing # of cores --> increase in lock wait time
***Writing to shared memory - increasing # of cores --> increase in wait for cache coherence protocol
***Competing for space in shared hardware cache - increasing # of cores --> increase in cache miss rate
***Competing for shared hardware resources - increasing # of cores --> increase in wait for resources
***Not enough tasks for cores --> idle cores

====Multicore packet processing: ''Section 4.2''====

====Sloppy counters: ''Section 4.3''====
Bottlenecks were encountered when the applications undergoing testing were referencing and updating shared counters for multiple cores. The solution in the paper is to use sloppy counters, which gets each core to track its own separate counts of references and uses a central shared counter to keep all counts on track. This is ideal because each core updates its counts by modifying its per-core counter, usually only needing access to its own local cache, cutting down on waiting for locks or serialization. Sloppy counters are also backwards-compatible with existing shared-counter code, making its implementation much easier to accomplish. The main disadvantages of the sloppy counters are that in situations where object de-allocation occurs often, because the de-allocation itself is an expensive operation, and the counters use up space proportional to the number of cores.

====Lock-free comparison: ''Section 4.4''====
This section describes a specific instance of unnecessary locking.

====Per-Core Data Structures: ''Section 4.5''====
Three centralized data structures were causing bottlenecks - a per-superblock list of open files, vfsmount table, the packet buffers free list. Each data structure was decentralized to per-core versions of itself. In the case of vfsmount the central data structure was maintained, and any per-core misses got written from the central table to the per-core table.

====Eliminating false sharing: ''Section 4.6''====
Misplaced variables on the cache cause different cores to request the same line to be read and written at the same time often enough to significantly impact performance. By moving the often written variable to another line the bottleneck was removed.

====Avoiding unnecessary locking: ''Section 4.7''====
Many locks/mutexes have special cases where they don't need to lock. Likewise mutexes can be split from locking the whole data structure to locking a part of it. Both these changes remove or reduce bottlenecks.

====Conclusion: ''Sections 6 & 7''====
Everything so far indicates that MOSBENCH application can scale to 48 cores. This scaling required a few modest changes to remove bottlenecks. The MIT team speculate that that trend will continue as the number of cores increase. They also state that things not bottlenecked by the CPU are harder to fix.

We can eliminate most kernel bottlenecks that the applications hits most often with minor changes. Most changes were well known methodology, with the exception of Sloppy counters. This study is limited by the removal of the IO bottleneck, but it does suggest that traditional implementations can be made scalable.

==Critique==
What is good and not-so-good about this paper? You may discuss both the style and content;
be sure to ground your discussion with specific references. Simple assertions that something is good or bad is not enough - you must explain why.

Since this is a "my implementation is better then your implementation" paper the "goodness" of content can be impartially determined by its fairness and the honesty of the authors.

====Content(Fairness): ''Section 5''====

=====memcached: ''Section 5.3''=====
memcached is treated with near perfect fairness in the paper. Its an in-memory service, so the ignored storage IO bottleneck does not affect it at all. Likewise the "stock" and "PK" implementations are given the same test suite, so there is no advantage given to either. memcached itself is non-scalable, so the MIT team was forced to run one instance per-core to keep up throughput. The FAQ at memcached.org's wiki suggests using multiple implementations per-server as a work around to another problem, which implies that running multiple instances of the server is the same, or nearly the same, as running one larger server [1]. In the end memcached was bottlenecked by the network card.

[1] memcached's wiki: http://code.google.com/p/memcached/wiki/FAQ#Can_I_use_different_size_caches_across_servers_and_will_memcache

=====Apache: ''Section 5.4''=====
Linux has a built in kernel flaw where network packets are forced to travel though multiple queues before they arrive at queue where they can be processed by the application. This imposes significant costs on multi-core systems due to queue locking costs. This flaw inherently diminishes the performance of Apache on multi-core system due to multiple threads spread across cores being forced to deal with these mutex (mutual exclusion) costs. For the sake of this experiment Apache had a separate instance on every core listening on different ports which is not a practical real world application but merely an attempt to implement better parallel execution on a traditional kernel. The patched kernel implementation of the network stack is also specific to the problem at hand, which is processing multiple short lived connections across multiple cores. Although this provides a performance increase in the given scenario, in more general applications network performance might suffer. These tests were also rigged to avoid bottlenecks in place by network and file storage hardware. Meaning, making the proposed modifications to the kernel wont necessarily produce the same increase in productivity as described in the article. This is very much evident in the test where performance degrades past 36 cores due to limitation of the networking hardware. ''Which is not a problem as the paper specifically states that there are hardware limitations.''

=====gmake: ''Section 5.6''=====
Since the inherent nature of gmake makes it quite parallel, the testing and updating that were attempted on gmake resulted in essentially the same scalability results for both the stock and modified kernel. The only change that was found was that gmake spent slightly less time at the system level because of the changes that were made to the system's caching. As stated in the paper, the execution time of gmake relies quite heavily on the compiler that is uses with gmake, so depending on which compiler was chosen, gmake could run worse or even slightly better. In any case, there seems to be no fairness concerns when it comes to the scalability testing of gmake as the same application load-out was used for all of the tests.

====Style====
Style Criterion (feel free to add I have no idea what should go here):
- does the paper present information out of order?
- does the paper present needless information?
- does the paper have any sections that are inherently confusing? Wrong? or use bad methodology?
- is the paper easy to read through, or does it change subjects repeatedly?
- does the paper have too many "long-winded" sentences, making it seem like the authors are just trying to add extra words to make it seem more important? - I think maybe limit this to run-on sentences.
- Check for grammar

===Conclusion===
[[Of the entire essay...]]

===References===
[1] Silas Boyd-Wickizer et al. "An Analysis of Linux Scalability to Many Cores". In ''OSDI '10, 9th USENIX Symposium on OS Design and Implementation'', Vancouver, BC, Canada, 2010. http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf.

You will almost certainly have to refer to other resources; please cite these resources in the style of citation of the papers assigned (inlined numbered references). Place your bibliographic entries in this section.

gmake:

[http://www.gnu.org/software/make/manual/make.html gmake Manual]

[http://www.gnu.org/software/make/ gmake Main Page]

===Deprecated===
====Background Concepts====
* Exim: ''Section 3.1'':
**Exim is a mail server for Unix. It's fairly parallel. The server forks a new process for each connection and twice to deliver each message. It spends 69% of its time in the kernel on a single core.

* PostgreSQL: ''Section 3.4'':
**As implied by the name PostgreSQL is a SQL database. PostgreSQL starts a separate process for each connection and uses kernel locking interfaces extensively to provide concurrent access to the database. Due to bottlenecks introduced in its code and in the kernel code, the amount of time PostgreSQL spends in the kernel increases very rapidly with addition of new cores. On a single core system PostgreSQL spends only 1.5% of its time in the kernel. On a 48 core system the execution time in the kernel jumps to 82%.

Talk:COMP 3000 Essay 2 2010 Question 1

2010-12-02T15:55:10Z

Abown: /* Research problem */

=Class and Notices=
(Nov. 30, 2010) Prof. Anil stated that we should focus on the 3 easiest to understand parts in section 5 and elaborate on them.

- Also, I, Daniel B., work Thursday night, so I will be finishing up as much of my part as I can for the essay before Thursday morning's class, then maybe we can all meet up in a lab in HP and put the finishing touches on the essay. I will be available online Wednesday night from about 6:30pm onwards and will be in the game dev lab or CCSS lounge Wednesday morning from about 11am to 2pm if anyone would like to meet up with me at those times.

- [[I suggest we meet up Thursday morning after Operating Systems in order to discuss and finalize the essay. Maybe we can even designate a lab for the group to meet up in. Any suggestions?]] - Daniel B.

- HP 3115 since there wont be a class in there (as its our tutorial and we know there won't be anyone there)
-- Go to Wireless Lab next to CCSS Lounge. Andrew and Dan B. will be there.

- If its all the same to you guys mind if I just join you via msn or iirc? Or phone if you really want. -Rannath

- I'm working today, but I'll be at a computer reading this page/contributing to my section. Depending on how busy I am, I should be able to get some significant writing in before 4pm today on my section and any additional sections required. RP

- I wont be there either. that does not mean i wont/cant contribute. I'll be on msn or you can just email me. -kirill

=Group members=
Patrick Young [mailto:Rannath@gmail.com Rannath@gmail.com]

Daniel Beimers [mailto:demongyro@gmail.com demongyro@gmail.com]

Andrew Bown [mailto:abown2@connect.carleton.ca abown2@connect.carleton.ca]

Kirill Kashigin [mailto:k.kashigin@gmail.com k.kashigin@gmail.com]

Rovic Perdon [mailto:rperdon@gmail.com rperdon@gmail.com]

Daniel Sont [mailto:dan.sont@gmail.com dan.sont@gmail.com]

=Methodology=
We should probably have our work verified by at least one group member before posting to the actual page

=To Do=
*Improve the grammar/structure of the paper section & add links to supplementary info
*Background Concepts -fill in info (fii)
*Contribution -fii
*Critique -fii
*References -fii

===Claim Sections===
* I claim Exim and memcached for background and critique -[[Rannath]]
* also per-core data structures, false sharing and unessesary locking for contribution -[[Rannath]]
* For starters I will take the Scalability Tutorial and gmake. Since the part for gmake is short in the paper, I will grab a few more sections later on. - [[Daniel B.]]
* Also, I will take sloppy counters as well - [[Daniel B.]]
* I'm gonna put some work into the apache and postgresql sections - kirill
* Just as a note Anil in class Thuesday the 30th of November said that we only need to explain 3 of the applications and not all 7 - [[Andrew]]
* I'll do the Research problem and contribution sections. - [[Andrew]]
* I will work on contribution - [[Rovic]]
* I'm gonna whip something up for 4.2 since there appears to be nothing mentioned about it. -kirill

* So here is the claims and unclaimed section. Add your name next to one if you want to take it on.
** gmake - Daniel B.
** memcached - Rannath
** Apache - Kirill
** [[(Exim, PostgreSQL, Metis, and Psearchy will not be needed as the professor said we only need to explain 3)]]
** Research Problem - Andrew
** Contribution - Rovic
** Critic, Style - Everyone
** Conclusion (also discussion) - Rannath, but I need someone to help flesh it out, I got the salient points down.
** References - Everyone
** Essay Conclusion - [[Nobody]]

=Essay=
===Paper===
This paper was authored by - Silas Boyd-Wickizer, Austin T. Clements, Yandong Mao, Aleksey Pesterev, M. Frans Kaashoek, Robert Morris, and Nickolai Zeldovich.

They all work at MIT CSAIL.

[http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf The paper: An Analysis of Linux Scalability to Many Cores]

===Background Concepts===
Explain briefly the background concepts and ideas that your fellow classmates will need to know first in order to understand your assigned paper.

Ideas to explain:
- thread (maybe)
- Linux's move towards scalability precedes this paper. (assert this, no explanation needed, maybe a few examples)
- Summarize scalability tutorial (Section 4.1 of the paper) focus on what makes something (non-)scalable
- Describe the programs tested (what they do, how they're programmed (serial vs parallel), where to the do their processing)

====memchached: ''Section 3.2''====
memcached is an in-memory hash table server. One instance running on many cores is bottlenecked by an internal lock. The MIT team ran one instance per-core to avoid the problem. Clients each connect to a single instance. This allows the server to simulate parallelism. With few requests, memcached spends 80% of its time in the kernel on one core, mostly processing packets.

====Apache: ''Section 3.3''====
Apache is a web server. In the case of this study, Apache has been configured to run a separate process on each core. Each process, in turn, has multiple threads (Making it a perfect example of parallel programming). One thread to service incoming connections and various other threads to service those connections. On a single core processor, Apache spends 60% of its execution time in the kernel.

====gmake: ''Section 3.5''====
gmake is an unofficial default benchmark in the Linux community which is used in this paper to build the Linux kernel. gmake is already quite parallel, creating more processes than cores, so that it can make proper use of multiple cores, and involves much reading and writing of files, as it is used to build the Linux kernel. gmake is limited in scalability due to the serial processes that run at the beginning and end of its execution. gmake spends much of its execution time with its compiler, but still spend 7.6% of its time in system time. [1]

==Research problem==
my references are just below because it is easier for numbering the data later.

As technological progress the number of core a main processor can have is increasing at an impressive rate. Soon personal computers will have so many cores that scalability will be an issue. There has to be a way that standard user level Linux kernel will scale with a 48-core system[1]. The problem with a standard Linux operating is they are not designed for massive scalability which will soon be a problem. The issue with scalability is that a solo core will perform much more work compared to a single core working with 47 other cores. Although traditional logic that situation makes sense because 48 cores are dividing the work. But when processing information a process the main goal is to finish so as long as possible every core should be doing a much work as possible.

To fix those scalability issues it is necessary to focus on three major areas: the Linux kernel, user level design and how application use of kernel services. The Linux kernel can be improved be to improve sharing and have the advantage of recent iterations are beginning to implement scalability features. On the user level design applications can be improved so that there is more focus on parallelism since some programs have not implements those improved features. The final aspect of improving scalability is how an application uses kernel services to share resources better so that different aspects of the program are not conflicting over the same services. All of the bottlenecks are found actually only take a little work to avoid.[1]

This research is based on much research which was created before in the development of scalability for UNIX system. The major developments from shared memory machines [2], wait-free synchronization to fast message passing have created a base set of techniques which can be used to improve scalability. These techniques have been incorporated in all major operation system including Linux, Mac OS X and Windows. Linux has been improved with kernel subsystems such as Read-Copy-Update which an algorithm for which is used to avoid locks and atomic instructions which lower scalability.[3] The is also an excellent base a research on Linux scalability studies to base this research paper. These paper include a on doing scalability on a 32-core machine. [4] That research can improve the results by learning from the experiments already performed by researchers. This research also aid identifying bottlenecks which speed up researching solutions for those bottlenecks.

[2] J. Kuskin, D. Ofelt, M. Heinrich, J. Heinlein, R. Simoni, K. Gharachorloo, J. Chapin, D. Nakahira, J. Baxter, M. Horowitz, A. Gupta, M. Rosenblum, and J. Hennessy. The Stanford FLASH multiprocessor. In Proc. of the 21st ISCA, pages 302–313,1994.

[3] P. E. McKenney, D. Sarma, A. Arcangeli, A. Kleen, O. Krieger, and R. Russell. Read-copy-update. In Proceedings of the Linux Symposium 2002, pages 338-367, Ottawa Ontario, June 2002

[4] C. Yan, Y. Chen, and S. Yuanchun. OSMark: A benchmark suite for understanding parallel scalability of operating systems on large scale multi-cores. In 2009 2nd International Conference on Computer Science and Information Technology, pages 313–317, 2009

==Contribution==
What was implemented? Why is it any better than what came before?

Summarize info from Section 4.2 onwards, maybe put graphs from Section 5 here to provide support for improvements (if that isn't unethical/illegal)?

- So long as we cite the paper and don't pretend the graphs are ours, we are ok, since we are writing an explanation/critic of the paper.

-I'm just using this as a notepad, do not copy/paste this section, I will put in a properly written set of paragraphs which will fit with the contribution questions asked. -RP

-==Work in Progress==-- -Rovic P.
This research contributes by evaluating the scalability discrepancies of applications programming and kernel programming. Key discoveries in this research show the effectiveness of the kernel in handling scaling amongst CPU cores. This has also shown that scaling in application programming should be more the focus. It has been shown that simple scaling techniques (list techniques) such as programming parallelism (look up more stuff to back this up and quotes). (Sloppy counter effectiveness, possible positive contributions, what has been used (internet search), what hasn’t been used.) Read conclusion, 2nd paragraph.

One reason the
required changes are modest is that stock Linux already
incorporates many modifications to improve scalability.
More speculatively, perhaps it is the case that Linux’s
system-call API is well suited to an implementation that
avoids unnecessary contention over kernel objects.

===Section 4.1 problems:===
**The percentage of serialization in a program has a lot to do with how much an application can be sped up. As from the example in the paper, it seems to follow Amdahl's law (e.g. 25% serialization --> limit of 4x speedup).
**Types of serializing interactions found in the MOSBENCH apps:
***Locking of shared data structure - increasing # of cores --> increase in lock wait time
***Writing to shared memory - increasing # of cores --> increase in wait for cache coherence protocol
***Competing for space in shared hardware cache - increasing # of cores --> increase in cache miss rate
***Competing for shared hardware resources - increasing # of cores --> increase in wait for resources
***Not enough tasks for cores --> idle cores

====Multicore packet processing: ''Section 4.2''====

====Sloppy counters: ''Section 4.3''====
Bottlenecks were encountered when the applications undergoing testing were referencing and updating shared counters for multiple cores. The solution in the paper is to use sloppy counters, which gets each core to track its own separate counts of references and uses a central shared counter to keep all counts on track. This is ideal because each core updates its counts by modifying its per-core counter, usually only needing access to its own local cache, cutting down on waiting for locks or serialization. Sloppy counters are also backwards-compatible with existing shared-counter code, making its implementation much easier to accomplish. The main disadvantages of the sloppy counters are that in situations where object de-allocation occurs often, because the de-allocation itself is an expensive operation, and the counters use up space proportional to the number of cores.

====Lock-free comparison: ''Section 4.4''====
This section describes a specific instance of unnecessary locking.

====Per-Core Data Structures: ''Section 4.5''====
Three centralized data structures were causing bottlenecks - a per-superblock list of open files, vfsmount table, the packet buffers free list. Each data structure was decentralized to per-core versions of itself. In the case of vfsmount the central data structure was maintained, and any per-core misses got written from the central table to the per-core table.

====Eliminating false sharing: ''Section 4.6''====
Misplaced variables on the cache cause different cores to request the same line to be read and written at the same time often enough to significantly impact performance. By moving the often written variable to another line the bottleneck was removed.

====Avoiding unnecessary locking: ''Section 4.7''====
Many locks/mutexes have special cases where they don't need to lock. Likewise mutexes can be split from locking the whole data structure to locking a part of it. Both these changes remove or reduce bottlenecks.

====Conclusion: ''Sections 6 & 7''====
Everything so far indicates that MOSBENCH application can scale to 48 cores. This scaling required a few modest changes to remove bottlenecks. The MIT team speculate that that trend will continue as the number of cores increase. They also state that things not bottlenecked by the CPU are harder to fix.

We can eliminate most kernel bottlenecks that the applications hits most often with minor changes. Most changes were well known methodology, with the exception of Sloppy counters. This study is limited by the removal of the IO bottleneck, but it does suggest that traditional implementations can be made scalable.

==Critique==
What is good and not-so-good about this paper? You may discuss both the style and content;
be sure to ground your discussion with specific references. Simple assertions that something is good or bad is not enough - you must explain why.

Since this is a "my implementation is better then your implementation" paper the "goodness" of content can be impartially determined by its fairness and the honesty of the authors.

====Content(Fairness): ''Section 5''====

=====memcached: ''Section 5.3''=====
memcached is treated with near perfect fairness in the paper. Its an in-memory service, so the ignored storage IO bottleneck does not affect it at all. Likewise the "stock" and "PK" implementations are given the same test suite, so there is no advantage given to either. memcached itself is non-scalable, so the MIT team was forced to run one instance per-core to keep up throughput. The FAQ at memcached.org's wiki suggests using multiple implementations per-server as a work around to another problem, which implies that running multiple instances of the server is the same, or nearly the same, as running one larger server [1]. In the end memcached was bottlenecked by the network card.

[1] memcached's wiki: http://code.google.com/p/memcached/wiki/FAQ#Can_I_use_different_size_caches_across_servers_and_will_memcache

=====Apache: ''Section 5.4''=====
Linux has a built in kernel flaw where network packets are forced to travel though multiple queues before they arrive at queue where they can be processed by the application. This imposes significant costs on multi-core systems due to queue locking costs. This flaw inherently diminishes the performance of Apache on multi-core system due to multiple threads spread across cores being forced to deal with these mutex (mutual exclusion) costs. For the sake of this experiment Apache had a separate instance on every core listening on different ports which is not a practical real world application but merely an attempt to implement better parallel execution on a traditional kernel. The patched kernel implementation of the network stack is also specific to the problem at hand, which is processing multiple short lived connections across multiple cores. Although this provides a performance increase in the given scenario, in more general applications network performance might suffer. These tests were also rigged to avoid bottlenecks in place by network and file storage hardware. Meaning, making the proposed modifications to the kernel wont necessarily produce the same increase in productivity as described in the article. This is very much evident in the test where performance degrades past 36 cores due to limitation of the networking hardware. ''Which is not a problem as the paper specifically states that there are hardware limitations.''

=====gmake: ''Section 5.6''=====
Since the inherent nature of gmake makes it quite parallel, the testing and updating that were attempted on gmake resulted in essentially the same scalability results for both the stock and modified kernel. The only change that was found was that gmake spent slightly less time at the system level because of the changes that were made to the system's caching. As stated in the paper, the execution time of gmake relies quite heavily on the compiler that is uses with gmake, so depending on which compiler was chosen, gmake could run worse or even slightly better. In any case, there seems to be no fairness concerns when it comes to the scalability testing of gmake as the same application load-out was used for all of the tests.

====Style====
Style Criterion (feel free to add I have no idea what should go here):
- does the paper present information out of order?
- does the paper present needless information?
- does the paper have any sections that are inherently confusing? Wrong? or use bad methodology?
- is the paper easy to read through, or does it change subjects repeatedly?
- does the paper have too many "long-winded" sentences, making it seem like the authors are just trying to add extra words to make it seem more important? - I think maybe limit this to run-on sentences.
- Check for grammar

===Conclusion===
[[Of the entire essay...]]

===References===
[1] Silas Boyd-Wickizer et al. "An Analysis of Linux Scalability to Many Cores". In ''OSDI '10, 9th USENIX Symposium on OS Design and Implementation'', Vancouver, BC, Canada, 2010. http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf.

You will almost certainly have to refer to other resources; please cite these resources in the style of citation of the papers assigned (inlined numbered references). Place your bibliographic entries in this section.

gmake:

[http://www.gnu.org/software/make/manual/make.html gmake Manual]

[http://www.gnu.org/software/make/ gmake Main Page]

===Deprecated===
====Background Concepts====
* Exim: ''Section 3.1'':
**Exim is a mail server for Unix. It's fairly parallel. The server forks a new process for each connection and twice to deliver each message. It spends 69% of its time in the kernel on a single core.

* PostgreSQL: ''Section 3.4'':
**As implied by the name PostgreSQL is a SQL database. PostgreSQL starts a separate process for each connection and uses kernel locking interfaces extensively to provide concurrent access to the database. Due to bottlenecks introduced in its code and in the kernel code, the amount of time PostgreSQL spends in the kernel increases very rapidly with addition of new cores. On a single core system PostgreSQL spends only 1.5% of its time in the kernel. On a 48 core system the execution time in the kernel jumps to 82%.

Talk:COMP 3000 Essay 2 2010 Question 1

2010-12-02T15:54:57Z

Abown: /* Contribution */

=Class and Notices=
(Nov. 30, 2010) Prof. Anil stated that we should focus on the 3 easiest to understand parts in section 5 and elaborate on them.

- Also, I, Daniel B., work Thursday night, so I will be finishing up as much of my part as I can for the essay before Thursday morning's class, then maybe we can all meet up in a lab in HP and put the finishing touches on the essay. I will be available online Wednesday night from about 6:30pm onwards and will be in the game dev lab or CCSS lounge Wednesday morning from about 11am to 2pm if anyone would like to meet up with me at those times.

- [[I suggest we meet up Thursday morning after Operating Systems in order to discuss and finalize the essay. Maybe we can even designate a lab for the group to meet up in. Any suggestions?]] - Daniel B.

- HP 3115 since there wont be a class in there (as its our tutorial and we know there won't be anyone there)
-- Go to Wireless Lab next to CCSS Lounge. Andrew and Dan B. will be there.

- If its all the same to you guys mind if I just join you via msn or iirc? Or phone if you really want. -Rannath

- I'm working today, but I'll be at a computer reading this page/contributing to my section. Depending on how busy I am, I should be able to get some significant writing in before 4pm today on my section and any additional sections required. RP

- I wont be there either. that does not mean i wont/cant contribute. I'll be on msn or you can just email me. -kirill

=Group members=
Patrick Young [mailto:Rannath@gmail.com Rannath@gmail.com]

Daniel Beimers [mailto:demongyro@gmail.com demongyro@gmail.com]

Andrew Bown [mailto:abown2@connect.carleton.ca abown2@connect.carleton.ca]

Kirill Kashigin [mailto:k.kashigin@gmail.com k.kashigin@gmail.com]

Rovic Perdon [mailto:rperdon@gmail.com rperdon@gmail.com]

Daniel Sont [mailto:dan.sont@gmail.com dan.sont@gmail.com]

=Methodology=
We should probably have our work verified by at least one group member before posting to the actual page

=To Do=
*Improve the grammar/structure of the paper section & add links to supplementary info
*Background Concepts -fill in info (fii)
*Contribution -fii
*Critique -fii
*References -fii

===Claim Sections===
* I claim Exim and memcached for background and critique -[[Rannath]]
* also per-core data structures, false sharing and unessesary locking for contribution -[[Rannath]]
* For starters I will take the Scalability Tutorial and gmake. Since the part for gmake is short in the paper, I will grab a few more sections later on. - [[Daniel B.]]
* Also, I will take sloppy counters as well - [[Daniel B.]]
* I'm gonna put some work into the apache and postgresql sections - kirill
* Just as a note Anil in class Thuesday the 30th of November said that we only need to explain 3 of the applications and not all 7 - [[Andrew]]
* I'll do the Research problem and contribution sections. - [[Andrew]]
* I will work on contribution - [[Rovic]]
* I'm gonna whip something up for 4.2 since there appears to be nothing mentioned about it. -kirill

* So here is the claims and unclaimed section. Add your name next to one if you want to take it on.
** gmake - Daniel B.
** memcached - Rannath
** Apache - Kirill
** [[(Exim, PostgreSQL, Metis, and Psearchy will not be needed as the professor said we only need to explain 3)]]
** Research Problem - Andrew
** Contribution - Rovic
** Critic, Style - Everyone
** Conclusion (also discussion) - Rannath, but I need someone to help flesh it out, I got the salient points down.
** References - Everyone
** Essay Conclusion - [[Nobody]]

=Essay=
===Paper===
This paper was authored by - Silas Boyd-Wickizer, Austin T. Clements, Yandong Mao, Aleksey Pesterev, M. Frans Kaashoek, Robert Morris, and Nickolai Zeldovich.

They all work at MIT CSAIL.

[http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf The paper: An Analysis of Linux Scalability to Many Cores]

===Background Concepts===
Explain briefly the background concepts and ideas that your fellow classmates will need to know first in order to understand your assigned paper.

Ideas to explain:
- thread (maybe)
- Linux's move towards scalability precedes this paper. (assert this, no explanation needed, maybe a few examples)
- Summarize scalability tutorial (Section 4.1 of the paper) focus on what makes something (non-)scalable
- Describe the programs tested (what they do, how they're programmed (serial vs parallel), where to the do their processing)

====memchached: ''Section 3.2''====
memcached is an in-memory hash table server. One instance running on many cores is bottlenecked by an internal lock. The MIT team ran one instance per-core to avoid the problem. Clients each connect to a single instance. This allows the server to simulate parallelism. With few requests, memcached spends 80% of its time in the kernel on one core, mostly processing packets.

====Apache: ''Section 3.3''====
Apache is a web server. In the case of this study, Apache has been configured to run a separate process on each core. Each process, in turn, has multiple threads (Making it a perfect example of parallel programming). One thread to service incoming connections and various other threads to service those connections. On a single core processor, Apache spends 60% of its execution time in the kernel.

====gmake: ''Section 3.5''====
gmake is an unofficial default benchmark in the Linux community which is used in this paper to build the Linux kernel. gmake is already quite parallel, creating more processes than cores, so that it can make proper use of multiple cores, and involves much reading and writing of files, as it is used to build the Linux kernel. gmake is limited in scalability due to the serial processes that run at the beginning and end of its execution. gmake spends much of its execution time with its compiler, but still spend 7.6% of its time in system time. [1]

===Research problem===
my references are just below because it is easier for numbering the data later.

As technological progress the number of core a main processor can have is increasing at an impressive rate. Soon personal computers will have so many cores that scalability will be an issue. There has to be a way that standard user level Linux kernel will scale with a 48-core system[1]. The problem with a standard Linux operating is they are not designed for massive scalability which will soon be a problem. The issue with scalability is that a solo core will perform much more work compared to a single core working with 47 other cores. Although traditional logic that situation makes sense because 48 cores are dividing the work. But when processing information a process the main goal is to finish so as long as possible every core should be doing a much work as possible.

To fix those scalability issues it is necessary to focus on three major areas: the Linux kernel, user level design and how application use of kernel services. The Linux kernel can be improved be to improve sharing and have the advantage of recent iterations are beginning to implement scalability features. On the user level design applications can be improved so that there is more focus on parallelism since some programs have not implements those improved features. The final aspect of improving scalability is how an application uses kernel services to share resources better so that different aspects of the program are not conflicting over the same services. All of the bottlenecks are found actually only take a little work to avoid.[1]

This research is based on much research which was created before in the development of scalability for UNIX system. The major developments from shared memory machines [2], wait-free synchronization to fast message passing have created a base set of techniques which can be used to improve scalability. These techniques have been incorporated in all major operation system including Linux, Mac OS X and Windows. Linux has been improved with kernel subsystems such as Read-Copy-Update which an algorithm for which is used to avoid locks and atomic instructions which lower scalability.[3] The is also an excellent base a research on Linux scalability studies to base this research paper. These paper include a on doing scalability on a 32-core machine. [4] That research can improve the results by learning from the experiments already performed by researchers. This research also aid identifying bottlenecks which speed up researching solutions for those bottlenecks.

[2] J. Kuskin, D. Ofelt, M. Heinrich, J. Heinlein, R. Simoni, K. Gharachorloo, J. Chapin, D. Nakahira, J. Baxter, M. Horowitz, A. Gupta, M. Rosenblum, and J. Hennessy. The Stanford FLASH multiprocessor. In Proc. of the 21st ISCA, pages 302–313,1994.

[3] P. E. McKenney, D. Sarma, A. Arcangeli, A. Kleen, O. Krieger, and R. Russell. Read-copy-update. In Proceedings of the Linux Symposium 2002, pages 338-367, Ottawa Ontario, June 2002

[4] C. Yan, Y. Chen, and S. Yuanchun. OSMark: A benchmark suite for understanding parallel scalability of operating systems on large scale multi-cores. In 2009 2nd International Conference on Computer Science and Information Technology, pages 313–317, 2009

==Contribution==
What was implemented? Why is it any better than what came before?

Summarize info from Section 4.2 onwards, maybe put graphs from Section 5 here to provide support for improvements (if that isn't unethical/illegal)?

- So long as we cite the paper and don't pretend the graphs are ours, we are ok, since we are writing an explanation/critic of the paper.

-I'm just using this as a notepad, do not copy/paste this section, I will put in a properly written set of paragraphs which will fit with the contribution questions asked. -RP

-==Work in Progress==-- -Rovic P.
This research contributes by evaluating the scalability discrepancies of applications programming and kernel programming. Key discoveries in this research show the effectiveness of the kernel in handling scaling amongst CPU cores. This has also shown that scaling in application programming should be more the focus. It has been shown that simple scaling techniques (list techniques) such as programming parallelism (look up more stuff to back this up and quotes). (Sloppy counter effectiveness, possible positive contributions, what has been used (internet search), what hasn’t been used.) Read conclusion, 2nd paragraph.

One reason the
required changes are modest is that stock Linux already
incorporates many modifications to improve scalability.
More speculatively, perhaps it is the case that Linux’s
system-call API is well suited to an implementation that
avoids unnecessary contention over kernel objects.

===Section 4.1 problems:===
**The percentage of serialization in a program has a lot to do with how much an application can be sped up. As from the example in the paper, it seems to follow Amdahl's law (e.g. 25% serialization --> limit of 4x speedup).
**Types of serializing interactions found in the MOSBENCH apps:
***Locking of shared data structure - increasing # of cores --> increase in lock wait time
***Writing to shared memory - increasing # of cores --> increase in wait for cache coherence protocol
***Competing for space in shared hardware cache - increasing # of cores --> increase in cache miss rate
***Competing for shared hardware resources - increasing # of cores --> increase in wait for resources
***Not enough tasks for cores --> idle cores

====Multicore packet processing: ''Section 4.2''====

====Sloppy counters: ''Section 4.3''====
Bottlenecks were encountered when the applications undergoing testing were referencing and updating shared counters for multiple cores. The solution in the paper is to use sloppy counters, which gets each core to track its own separate counts of references and uses a central shared counter to keep all counts on track. This is ideal because each core updates its counts by modifying its per-core counter, usually only needing access to its own local cache, cutting down on waiting for locks or serialization. Sloppy counters are also backwards-compatible with existing shared-counter code, making its implementation much easier to accomplish. The main disadvantages of the sloppy counters are that in situations where object de-allocation occurs often, because the de-allocation itself is an expensive operation, and the counters use up space proportional to the number of cores.

====Lock-free comparison: ''Section 4.4''====
This section describes a specific instance of unnecessary locking.

====Per-Core Data Structures: ''Section 4.5''====
Three centralized data structures were causing bottlenecks - a per-superblock list of open files, vfsmount table, the packet buffers free list. Each data structure was decentralized to per-core versions of itself. In the case of vfsmount the central data structure was maintained, and any per-core misses got written from the central table to the per-core table.

====Eliminating false sharing: ''Section 4.6''====
Misplaced variables on the cache cause different cores to request the same line to be read and written at the same time often enough to significantly impact performance. By moving the often written variable to another line the bottleneck was removed.

====Avoiding unnecessary locking: ''Section 4.7''====
Many locks/mutexes have special cases where they don't need to lock. Likewise mutexes can be split from locking the whole data structure to locking a part of it. Both these changes remove or reduce bottlenecks.

====Conclusion: ''Sections 6 & 7''====
Everything so far indicates that MOSBENCH application can scale to 48 cores. This scaling required a few modest changes to remove bottlenecks. The MIT team speculate that that trend will continue as the number of cores increase. They also state that things not bottlenecked by the CPU are harder to fix.

We can eliminate most kernel bottlenecks that the applications hits most often with minor changes. Most changes were well known methodology, with the exception of Sloppy counters. This study is limited by the removal of the IO bottleneck, but it does suggest that traditional implementations can be made scalable.

==Critique==
What is good and not-so-good about this paper? You may discuss both the style and content;
be sure to ground your discussion with specific references. Simple assertions that something is good or bad is not enough - you must explain why.

Since this is a "my implementation is better then your implementation" paper the "goodness" of content can be impartially determined by its fairness and the honesty of the authors.

====Content(Fairness): ''Section 5''====

=====memcached: ''Section 5.3''=====
memcached is treated with near perfect fairness in the paper. Its an in-memory service, so the ignored storage IO bottleneck does not affect it at all. Likewise the "stock" and "PK" implementations are given the same test suite, so there is no advantage given to either. memcached itself is non-scalable, so the MIT team was forced to run one instance per-core to keep up throughput. The FAQ at memcached.org's wiki suggests using multiple implementations per-server as a work around to another problem, which implies that running multiple instances of the server is the same, or nearly the same, as running one larger server [1]. In the end memcached was bottlenecked by the network card.

[1] memcached's wiki: http://code.google.com/p/memcached/wiki/FAQ#Can_I_use_different_size_caches_across_servers_and_will_memcache

=====Apache: ''Section 5.4''=====
Linux has a built in kernel flaw where network packets are forced to travel though multiple queues before they arrive at queue where they can be processed by the application. This imposes significant costs on multi-core systems due to queue locking costs. This flaw inherently diminishes the performance of Apache on multi-core system due to multiple threads spread across cores being forced to deal with these mutex (mutual exclusion) costs. For the sake of this experiment Apache had a separate instance on every core listening on different ports which is not a practical real world application but merely an attempt to implement better parallel execution on a traditional kernel. The patched kernel implementation of the network stack is also specific to the problem at hand, which is processing multiple short lived connections across multiple cores. Although this provides a performance increase in the given scenario, in more general applications network performance might suffer. These tests were also rigged to avoid bottlenecks in place by network and file storage hardware. Meaning, making the proposed modifications to the kernel wont necessarily produce the same increase in productivity as described in the article. This is very much evident in the test where performance degrades past 36 cores due to limitation of the networking hardware. ''Which is not a problem as the paper specifically states that there are hardware limitations.''

=====gmake: ''Section 5.6''=====
Since the inherent nature of gmake makes it quite parallel, the testing and updating that were attempted on gmake resulted in essentially the same scalability results for both the stock and modified kernel. The only change that was found was that gmake spent slightly less time at the system level because of the changes that were made to the system's caching. As stated in the paper, the execution time of gmake relies quite heavily on the compiler that is uses with gmake, so depending on which compiler was chosen, gmake could run worse or even slightly better. In any case, there seems to be no fairness concerns when it comes to the scalability testing of gmake as the same application load-out was used for all of the tests.

====Style====
Style Criterion (feel free to add I have no idea what should go here):
- does the paper present information out of order?
- does the paper present needless information?
- does the paper have any sections that are inherently confusing? Wrong? or use bad methodology?
- is the paper easy to read through, or does it change subjects repeatedly?
- does the paper have too many "long-winded" sentences, making it seem like the authors are just trying to add extra words to make it seem more important? - I think maybe limit this to run-on sentences.
- Check for grammar

===Conclusion===
[[Of the entire essay...]]

===References===
[1] Silas Boyd-Wickizer et al. "An Analysis of Linux Scalability to Many Cores". In ''OSDI '10, 9th USENIX Symposium on OS Design and Implementation'', Vancouver, BC, Canada, 2010. http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf.

You will almost certainly have to refer to other resources; please cite these resources in the style of citation of the papers assigned (inlined numbered references). Place your bibliographic entries in this section.

gmake:

[http://www.gnu.org/software/make/manual/make.html gmake Manual]

[http://www.gnu.org/software/make/ gmake Main Page]

===Deprecated===
====Background Concepts====
* Exim: ''Section 3.1'':
**Exim is a mail server for Unix. It's fairly parallel. The server forks a new process for each connection and twice to deliver each message. It spends 69% of its time in the kernel on a single core.

* PostgreSQL: ''Section 3.4'':
**As implied by the name PostgreSQL is a SQL database. PostgreSQL starts a separate process for each connection and uses kernel locking interfaces extensively to provide concurrent access to the database. Due to bottlenecks introduced in its code and in the kernel code, the amount of time PostgreSQL spends in the kernel increases very rapidly with addition of new cores. On a single core system PostgreSQL spends only 1.5% of its time in the kernel. On a 48 core system the execution time in the kernel jumps to 82%.

Talk:COMP 3000 Essay 2 2010 Question 1

2010-12-02T15:54:28Z

Abown: /* Section 4.1 problems: */

=Class and Notices=
(Nov. 30, 2010) Prof. Anil stated that we should focus on the 3 easiest to understand parts in section 5 and elaborate on them.

- Also, I, Daniel B., work Thursday night, so I will be finishing up as much of my part as I can for the essay before Thursday morning's class, then maybe we can all meet up in a lab in HP and put the finishing touches on the essay. I will be available online Wednesday night from about 6:30pm onwards and will be in the game dev lab or CCSS lounge Wednesday morning from about 11am to 2pm if anyone would like to meet up with me at those times.

- [[I suggest we meet up Thursday morning after Operating Systems in order to discuss and finalize the essay. Maybe we can even designate a lab for the group to meet up in. Any suggestions?]] - Daniel B.

- HP 3115 since there wont be a class in there (as its our tutorial and we know there won't be anyone there)
-- Go to Wireless Lab next to CCSS Lounge. Andrew and Dan B. will be there.

- If its all the same to you guys mind if I just join you via msn or iirc? Or phone if you really want. -Rannath

- I'm working today, but I'll be at a computer reading this page/contributing to my section. Depending on how busy I am, I should be able to get some significant writing in before 4pm today on my section and any additional sections required. RP

- I wont be there either. that does not mean i wont/cant contribute. I'll be on msn or you can just email me. -kirill

=Group members=
Patrick Young [mailto:Rannath@gmail.com Rannath@gmail.com]

Daniel Beimers [mailto:demongyro@gmail.com demongyro@gmail.com]

Andrew Bown [mailto:abown2@connect.carleton.ca abown2@connect.carleton.ca]

Kirill Kashigin [mailto:k.kashigin@gmail.com k.kashigin@gmail.com]

Rovic Perdon [mailto:rperdon@gmail.com rperdon@gmail.com]

Daniel Sont [mailto:dan.sont@gmail.com dan.sont@gmail.com]

=Methodology=
We should probably have our work verified by at least one group member before posting to the actual page

=To Do=
*Improve the grammar/structure of the paper section & add links to supplementary info
*Background Concepts -fill in info (fii)
*Contribution -fii
*Critique -fii
*References -fii

===Claim Sections===
* I claim Exim and memcached for background and critique -[[Rannath]]
* also per-core data structures, false sharing and unessesary locking for contribution -[[Rannath]]
* For starters I will take the Scalability Tutorial and gmake. Since the part for gmake is short in the paper, I will grab a few more sections later on. - [[Daniel B.]]
* Also, I will take sloppy counters as well - [[Daniel B.]]
* I'm gonna put some work into the apache and postgresql sections - kirill
* Just as a note Anil in class Thuesday the 30th of November said that we only need to explain 3 of the applications and not all 7 - [[Andrew]]
* I'll do the Research problem and contribution sections. - [[Andrew]]
* I will work on contribution - [[Rovic]]
* I'm gonna whip something up for 4.2 since there appears to be nothing mentioned about it. -kirill

* So here is the claims and unclaimed section. Add your name next to one if you want to take it on.
** gmake - Daniel B.
** memcached - Rannath
** Apache - Kirill
** [[(Exim, PostgreSQL, Metis, and Psearchy will not be needed as the professor said we only need to explain 3)]]
** Research Problem - Andrew
** Contribution - Rovic
** Critic, Style - Everyone
** Conclusion (also discussion) - Rannath, but I need someone to help flesh it out, I got the salient points down.
** References - Everyone
** Essay Conclusion - [[Nobody]]

=Essay=
===Paper===
This paper was authored by - Silas Boyd-Wickizer, Austin T. Clements, Yandong Mao, Aleksey Pesterev, M. Frans Kaashoek, Robert Morris, and Nickolai Zeldovich.

They all work at MIT CSAIL.

[http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf The paper: An Analysis of Linux Scalability to Many Cores]

===Background Concepts===
Explain briefly the background concepts and ideas that your fellow classmates will need to know first in order to understand your assigned paper.

Ideas to explain:
- thread (maybe)
- Linux's move towards scalability precedes this paper. (assert this, no explanation needed, maybe a few examples)
- Summarize scalability tutorial (Section 4.1 of the paper) focus on what makes something (non-)scalable
- Describe the programs tested (what they do, how they're programmed (serial vs parallel), where to the do their processing)

====memchached: ''Section 3.2''====
memcached is an in-memory hash table server. One instance running on many cores is bottlenecked by an internal lock. The MIT team ran one instance per-core to avoid the problem. Clients each connect to a single instance. This allows the server to simulate parallelism. With few requests, memcached spends 80% of its time in the kernel on one core, mostly processing packets.

====Apache: ''Section 3.3''====
Apache is a web server. In the case of this study, Apache has been configured to run a separate process on each core. Each process, in turn, has multiple threads (Making it a perfect example of parallel programming). One thread to service incoming connections and various other threads to service those connections. On a single core processor, Apache spends 60% of its execution time in the kernel.

====gmake: ''Section 3.5''====
gmake is an unofficial default benchmark in the Linux community which is used in this paper to build the Linux kernel. gmake is already quite parallel, creating more processes than cores, so that it can make proper use of multiple cores, and involves much reading and writing of files, as it is used to build the Linux kernel. gmake is limited in scalability due to the serial processes that run at the beginning and end of its execution. gmake spends much of its execution time with its compiler, but still spend 7.6% of its time in system time. [1]

===Research problem===
my references are just below because it is easier for numbering the data later.

As technological progress the number of core a main processor can have is increasing at an impressive rate. Soon personal computers will have so many cores that scalability will be an issue. There has to be a way that standard user level Linux kernel will scale with a 48-core system[1]. The problem with a standard Linux operating is they are not designed for massive scalability which will soon be a problem. The issue with scalability is that a solo core will perform much more work compared to a single core working with 47 other cores. Although traditional logic that situation makes sense because 48 cores are dividing the work. But when processing information a process the main goal is to finish so as long as possible every core should be doing a much work as possible.

To fix those scalability issues it is necessary to focus on three major areas: the Linux kernel, user level design and how application use of kernel services. The Linux kernel can be improved be to improve sharing and have the advantage of recent iterations are beginning to implement scalability features. On the user level design applications can be improved so that there is more focus on parallelism since some programs have not implements those improved features. The final aspect of improving scalability is how an application uses kernel services to share resources better so that different aspects of the program are not conflicting over the same services. All of the bottlenecks are found actually only take a little work to avoid.[1]

This research is based on much research which was created before in the development of scalability for UNIX system. The major developments from shared memory machines [2], wait-free synchronization to fast message passing have created a base set of techniques which can be used to improve scalability. These techniques have been incorporated in all major operation system including Linux, Mac OS X and Windows. Linux has been improved with kernel subsystems such as Read-Copy-Update which an algorithm for which is used to avoid locks and atomic instructions which lower scalability.[3] The is also an excellent base a research on Linux scalability studies to base this research paper. These paper include a on doing scalability on a 32-core machine. [4] That research can improve the results by learning from the experiments already performed by researchers. This research also aid identifying bottlenecks which speed up researching solutions for those bottlenecks.

[2] J. Kuskin, D. Ofelt, M. Heinrich, J. Heinlein, R. Simoni, K. Gharachorloo, J. Chapin, D. Nakahira, J. Baxter, M. Horowitz, A. Gupta, M. Rosenblum, and J. Hennessy. The Stanford FLASH multiprocessor. In Proc. of the 21st ISCA, pages 302–313,1994.

[3] P. E. McKenney, D. Sarma, A. Arcangeli, A. Kleen, O. Krieger, and R. Russell. Read-copy-update. In Proceedings of the Linux Symposium 2002, pages 338-367, Ottawa Ontario, June 2002

[4] C. Yan, Y. Chen, and S. Yuanchun. OSMark: A benchmark suite for understanding parallel scalability of operating systems on large scale multi-cores. In 2009 2nd International Conference on Computer Science and Information Technology, pages 313–317, 2009

==Contribution==
What was implemented? Why is it any better than what came before?

Summarize info from Section 4.2 onwards, maybe put graphs from Section 5 here to provide support for improvements (if that isn't unethical/illegal)?

- So long as we cite the paper and don't pretend the graphs are ours, we are ok, since we are writing an explanation/critic of the paper.

-I'm just using this as a notepad, do not copy/paste this section, I will put in a properly written set of paragraphs which will fit with the contribution questions asked. -RP

-==Work in Progress==-- -Rovic P.
This research contributes by evaluating the scalability discrepancies of applications programming and kernel programming. Key discoveries in this research show the effectiveness of the kernel in handling scaling amongst CPU cores. This has also shown that scaling in application programming should be more the focus. It has been shown that simple scaling techniques (list techniques) such as programming parallelism (look up more stuff to back this up and quotes). (Sloppy counter effectiveness, possible positive contributions, what has been used (internet search), what hasn’t been used.) Read conclusion, 2nd paragraph.

One reason the
required changes are modest is that stock Linux already
incorporates many modifications to improve scalability.
More speculatively, perhaps it is the case that Linux’s
system-call API is well suited to an implementation that
avoids unnecessary contention over kernel objects.

====Multicore packet processing: ''Section 4.2''====

====Sloppy counters: ''Section 4.3''====
Bottlenecks were encountered when the applications undergoing testing were referencing and updating shared counters for multiple cores. The solution in the paper is to use sloppy counters, which gets each core to track its own separate counts of references and uses a central shared counter to keep all counts on track. This is ideal because each core updates its counts by modifying its per-core counter, usually only needing access to its own local cache, cutting down on waiting for locks or serialization. Sloppy counters are also backwards-compatible with existing shared-counter code, making its implementation much easier to accomplish. The main disadvantages of the sloppy counters are that in situations where object de-allocation occurs often, because the de-allocation itself is an expensive operation, and the counters use up space proportional to the number of cores.

====Lock-free comparison: ''Section 4.4''====
This section describes a specific instance of unnecessary locking.

====Per-Core Data Structures: ''Section 4.5''====
Three centralized data structures were causing bottlenecks - a per-superblock list of open files, vfsmount table, the packet buffers free list. Each data structure was decentralized to per-core versions of itself. In the case of vfsmount the central data structure was maintained, and any per-core misses got written from the central table to the per-core table.

====Eliminating false sharing: ''Section 4.6''====
Misplaced variables on the cache cause different cores to request the same line to be read and written at the same time often enough to significantly impact performance. By moving the often written variable to another line the bottleneck was removed.

====Avoiding unnecessary locking: ''Section 4.7''====
Many locks/mutexes have special cases where they don't need to lock. Likewise mutexes can be split from locking the whole data structure to locking a part of it. Both these changes remove or reduce bottlenecks.

====Conclusion: ''Sections 6 & 7''====
Everything so far indicates that MOSBENCH application can scale to 48 cores. This scaling required a few modest changes to remove bottlenecks. The MIT team speculate that that trend will continue as the number of cores increase. They also state that things not bottlenecked by the CPU are harder to fix.

We can eliminate most kernel bottlenecks that the applications hits most often with minor changes. Most changes were well known methodology, with the exception of Sloppy counters. This study is limited by the removal of the IO bottleneck, but it does suggest that traditional implementations can be made scalable.

==Critique==
What is good and not-so-good about this paper? You may discuss both the style and content;
be sure to ground your discussion with specific references. Simple assertions that something is good or bad is not enough - you must explain why.

Since this is a "my implementation is better then your implementation" paper the "goodness" of content can be impartially determined by its fairness and the honesty of the authors.

====Content(Fairness): ''Section 5''====

=====memcached: ''Section 5.3''=====
memcached is treated with near perfect fairness in the paper. Its an in-memory service, so the ignored storage IO bottleneck does not affect it at all. Likewise the "stock" and "PK" implementations are given the same test suite, so there is no advantage given to either. memcached itself is non-scalable, so the MIT team was forced to run one instance per-core to keep up throughput. The FAQ at memcached.org's wiki suggests using multiple implementations per-server as a work around to another problem, which implies that running multiple instances of the server is the same, or nearly the same, as running one larger server [1]. In the end memcached was bottlenecked by the network card.

[1] memcached's wiki: http://code.google.com/p/memcached/wiki/FAQ#Can_I_use_different_size_caches_across_servers_and_will_memcache

=====Apache: ''Section 5.4''=====
Linux has a built in kernel flaw where network packets are forced to travel though multiple queues before they arrive at queue where they can be processed by the application. This imposes significant costs on multi-core systems due to queue locking costs. This flaw inherently diminishes the performance of Apache on multi-core system due to multiple threads spread across cores being forced to deal with these mutex (mutual exclusion) costs. For the sake of this experiment Apache had a separate instance on every core listening on different ports which is not a practical real world application but merely an attempt to implement better parallel execution on a traditional kernel. The patched kernel implementation of the network stack is also specific to the problem at hand, which is processing multiple short lived connections across multiple cores. Although this provides a performance increase in the given scenario, in more general applications network performance might suffer. These tests were also rigged to avoid bottlenecks in place by network and file storage hardware. Meaning, making the proposed modifications to the kernel wont necessarily produce the same increase in productivity as described in the article. This is very much evident in the test where performance degrades past 36 cores due to limitation of the networking hardware. ''Which is not a problem as the paper specifically states that there are hardware limitations.''

=====gmake: ''Section 5.6''=====
Since the inherent nature of gmake makes it quite parallel, the testing and updating that were attempted on gmake resulted in essentially the same scalability results for both the stock and modified kernel. The only change that was found was that gmake spent slightly less time at the system level because of the changes that were made to the system's caching. As stated in the paper, the execution time of gmake relies quite heavily on the compiler that is uses with gmake, so depending on which compiler was chosen, gmake could run worse or even slightly better. In any case, there seems to be no fairness concerns when it comes to the scalability testing of gmake as the same application load-out was used for all of the tests.

====Style====
Style Criterion (feel free to add I have no idea what should go here):
- does the paper present information out of order?
- does the paper present needless information?
- does the paper have any sections that are inherently confusing? Wrong? or use bad methodology?
- is the paper easy to read through, or does it change subjects repeatedly?
- does the paper have too many "long-winded" sentences, making it seem like the authors are just trying to add extra words to make it seem more important? - I think maybe limit this to run-on sentences.
- Check for grammar

===Conclusion===
[[Of the entire essay...]]

===References===
[1] Silas Boyd-Wickizer et al. "An Analysis of Linux Scalability to Many Cores". In ''OSDI '10, 9th USENIX Symposium on OS Design and Implementation'', Vancouver, BC, Canada, 2010. http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf.

You will almost certainly have to refer to other resources; please cite these resources in the style of citation of the papers assigned (inlined numbered references). Place your bibliographic entries in this section.

gmake:

[http://www.gnu.org/software/make/manual/make.html gmake Manual]

[http://www.gnu.org/software/make/ gmake Main Page]

===Deprecated===
====Background Concepts====
* Exim: ''Section 3.1'':
**Exim is a mail server for Unix. It's fairly parallel. The server forks a new process for each connection and twice to deliver each message. It spends 69% of its time in the kernel on a single core.

* PostgreSQL: ''Section 3.4'':
**As implied by the name PostgreSQL is a SQL database. PostgreSQL starts a separate process for each connection and uses kernel locking interfaces extensively to provide concurrent access to the database. Due to bottlenecks introduced in its code and in the kernel code, the amount of time PostgreSQL spends in the kernel increases very rapidly with addition of new cores. On a single core system PostgreSQL spends only 1.5% of its time in the kernel. On a 48 core system the execution time in the kernel jumps to 82%.

* Psearchy: ''Section 3.6''

* Metis: ''Section 3.7''

Talk:COMP 3000 Essay 2 2010 Question 1

2010-12-02T15:54:01Z

Abown: /* Contribution */

=Class and Notices=
(Nov. 30, 2010) Prof. Anil stated that we should focus on the 3 easiest to understand parts in section 5 and elaborate on them.

- Also, I, Daniel B., work Thursday night, so I will be finishing up as much of my part as I can for the essay before Thursday morning's class, then maybe we can all meet up in a lab in HP and put the finishing touches on the essay. I will be available online Wednesday night from about 6:30pm onwards and will be in the game dev lab or CCSS lounge Wednesday morning from about 11am to 2pm if anyone would like to meet up with me at those times.

- [[I suggest we meet up Thursday morning after Operating Systems in order to discuss and finalize the essay. Maybe we can even designate a lab for the group to meet up in. Any suggestions?]] - Daniel B.

- HP 3115 since there wont be a class in there (as its our tutorial and we know there won't be anyone there)
-- Go to Wireless Lab next to CCSS Lounge. Andrew and Dan B. will be there.

- If its all the same to you guys mind if I just join you via msn or iirc? Or phone if you really want. -Rannath

- I'm working today, but I'll be at a computer reading this page/contributing to my section. Depending on how busy I am, I should be able to get some significant writing in before 4pm today on my section and any additional sections required. RP

- I wont be there either. that does not mean i wont/cant contribute. I'll be on msn or you can just email me. -kirill

=Group members=
Patrick Young [mailto:Rannath@gmail.com Rannath@gmail.com]

Daniel Beimers [mailto:demongyro@gmail.com demongyro@gmail.com]

Andrew Bown [mailto:abown2@connect.carleton.ca abown2@connect.carleton.ca]

Kirill Kashigin [mailto:k.kashigin@gmail.com k.kashigin@gmail.com]

Rovic Perdon [mailto:rperdon@gmail.com rperdon@gmail.com]

Daniel Sont [mailto:dan.sont@gmail.com dan.sont@gmail.com]

=Methodology=
We should probably have our work verified by at least one group member before posting to the actual page

=To Do=
*Improve the grammar/structure of the paper section & add links to supplementary info
*Background Concepts -fill in info (fii)
*Contribution -fii
*Critique -fii
*References -fii

===Claim Sections===
* I claim Exim and memcached for background and critique -[[Rannath]]
* also per-core data structures, false sharing and unessesary locking for contribution -[[Rannath]]
* For starters I will take the Scalability Tutorial and gmake. Since the part for gmake is short in the paper, I will grab a few more sections later on. - [[Daniel B.]]
* Also, I will take sloppy counters as well - [[Daniel B.]]
* I'm gonna put some work into the apache and postgresql sections - kirill
* Just as a note Anil in class Thuesday the 30th of November said that we only need to explain 3 of the applications and not all 7 - [[Andrew]]
* I'll do the Research problem and contribution sections. - [[Andrew]]
* I will work on contribution - [[Rovic]]
* I'm gonna whip something up for 4.2 since there appears to be nothing mentioned about it. -kirill

* So here is the claims and unclaimed section. Add your name next to one if you want to take it on.
** gmake - Daniel B.
** memcached - Rannath
** Apache - Kirill
** [[(Exim, PostgreSQL, Metis, and Psearchy will not be needed as the professor said we only need to explain 3)]]
** Research Problem - Andrew
** Contribution - Rovic
** Critic, Style - Everyone
** Conclusion (also discussion) - Rannath, but I need someone to help flesh it out, I got the salient points down.
** References - Everyone
** Essay Conclusion - [[Nobody]]

=Essay=
===Paper===
This paper was authored by - Silas Boyd-Wickizer, Austin T. Clements, Yandong Mao, Aleksey Pesterev, M. Frans Kaashoek, Robert Morris, and Nickolai Zeldovich.

They all work at MIT CSAIL.

[http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf The paper: An Analysis of Linux Scalability to Many Cores]

===Background Concepts===
Explain briefly the background concepts and ideas that your fellow classmates will need to know first in order to understand your assigned paper.

Ideas to explain:
- thread (maybe)
- Linux's move towards scalability precedes this paper. (assert this, no explanation needed, maybe a few examples)
- Summarize scalability tutorial (Section 4.1 of the paper) focus on what makes something (non-)scalable
- Describe the programs tested (what they do, how they're programmed (serial vs parallel), where to the do their processing)

====memchached: ''Section 3.2''====
memcached is an in-memory hash table server. One instance running on many cores is bottlenecked by an internal lock. The MIT team ran one instance per-core to avoid the problem. Clients each connect to a single instance. This allows the server to simulate parallelism. With few requests, memcached spends 80% of its time in the kernel on one core, mostly processing packets.

====Apache: ''Section 3.3''====
Apache is a web server. In the case of this study, Apache has been configured to run a separate process on each core. Each process, in turn, has multiple threads (Making it a perfect example of parallel programming). One thread to service incoming connections and various other threads to service those connections. On a single core processor, Apache spends 60% of its execution time in the kernel.

====gmake: ''Section 3.5''====
gmake is an unofficial default benchmark in the Linux community which is used in this paper to build the Linux kernel. gmake is already quite parallel, creating more processes than cores, so that it can make proper use of multiple cores, and involves much reading and writing of files, as it is used to build the Linux kernel. gmake is limited in scalability due to the serial processes that run at the beginning and end of its execution. gmake spends much of its execution time with its compiler, but still spend 7.6% of its time in system time. [1]

===Research problem===
my references are just below because it is easier for numbering the data later.

As technological progress the number of core a main processor can have is increasing at an impressive rate. Soon personal computers will have so many cores that scalability will be an issue. There has to be a way that standard user level Linux kernel will scale with a 48-core system[1]. The problem with a standard Linux operating is they are not designed for massive scalability which will soon be a problem. The issue with scalability is that a solo core will perform much more work compared to a single core working with 47 other cores. Although traditional logic that situation makes sense because 48 cores are dividing the work. But when processing information a process the main goal is to finish so as long as possible every core should be doing a much work as possible.

To fix those scalability issues it is necessary to focus on three major areas: the Linux kernel, user level design and how application use of kernel services. The Linux kernel can be improved be to improve sharing and have the advantage of recent iterations are beginning to implement scalability features. On the user level design applications can be improved so that there is more focus on parallelism since some programs have not implements those improved features. The final aspect of improving scalability is how an application uses kernel services to share resources better so that different aspects of the program are not conflicting over the same services. All of the bottlenecks are found actually only take a little work to avoid.[1]

This research is based on much research which was created before in the development of scalability for UNIX system. The major developments from shared memory machines [2], wait-free synchronization to fast message passing have created a base set of techniques which can be used to improve scalability. These techniques have been incorporated in all major operation system including Linux, Mac OS X and Windows. Linux has been improved with kernel subsystems such as Read-Copy-Update which an algorithm for which is used to avoid locks and atomic instructions which lower scalability.[3] The is also an excellent base a research on Linux scalability studies to base this research paper. These paper include a on doing scalability on a 32-core machine. [4] That research can improve the results by learning from the experiments already performed by researchers. This research also aid identifying bottlenecks which speed up researching solutions for those bottlenecks.

[2] J. Kuskin, D. Ofelt, M. Heinrich, J. Heinlein, R. Simoni, K. Gharachorloo, J. Chapin, D. Nakahira, J. Baxter, M. Horowitz, A. Gupta, M. Rosenblum, and J. Hennessy. The Stanford FLASH multiprocessor. In Proc. of the 21st ISCA, pages 302–313,1994.

[3] P. E. McKenney, D. Sarma, A. Arcangeli, A. Kleen, O. Krieger, and R. Russell. Read-copy-update. In Proceedings of the Linux Symposium 2002, pages 338-367, Ottawa Ontario, June 2002

[4] C. Yan, Y. Chen, and S. Yuanchun. OSMark: A benchmark suite for understanding parallel scalability of operating systems on large scale multi-cores. In 2009 2nd International Conference on Computer Science and Information Technology, pages 313–317, 2009

==Section 4.1 problems:==
**The percentage of serialization in a program has a lot to do with how much an application can be sped up. As from the example in the paper, it seems to follow Amdahl's law (e.g. 25% serialization --> limit of 4x speedup).
**Types of serializing interactions found in the MOSBENCH apps:
***Locking of shared data structure - increasing # of cores --> increase in lock wait time
***Writing to shared memory - increasing # of cores --> increase in wait for cache coherence protocol
***Competing for space in shared hardware cache - increasing # of cores --> increase in cache miss rate
***Competing for shared hardware resources - increasing # of cores --> increase in wait for resources
***Not enough tasks for cores --> idle cores

==Contribution==
What was implemented? Why is it any better than what came before?

Summarize info from Section 4.2 onwards, maybe put graphs from Section 5 here to provide support for improvements (if that isn't unethical/illegal)?

- So long as we cite the paper and don't pretend the graphs are ours, we are ok, since we are writing an explanation/critic of the paper.

-I'm just using this as a notepad, do not copy/paste this section, I will put in a properly written set of paragraphs which will fit with the contribution questions asked. -RP

-==Work in Progress==-- -Rovic P.
This research contributes by evaluating the scalability discrepancies of applications programming and kernel programming. Key discoveries in this research show the effectiveness of the kernel in handling scaling amongst CPU cores. This has also shown that scaling in application programming should be more the focus. It has been shown that simple scaling techniques (list techniques) such as programming parallelism (look up more stuff to back this up and quotes). (Sloppy counter effectiveness, possible positive contributions, what has been used (internet search), what hasn’t been used.) Read conclusion, 2nd paragraph.

One reason the
required changes are modest is that stock Linux already
incorporates many modifications to improve scalability.
More speculatively, perhaps it is the case that Linux’s
system-call API is well suited to an implementation that
avoids unnecessary contention over kernel objects.

====Multicore packet processing: ''Section 4.2''====

====Sloppy counters: ''Section 4.3''====
Bottlenecks were encountered when the applications undergoing testing were referencing and updating shared counters for multiple cores. The solution in the paper is to use sloppy counters, which gets each core to track its own separate counts of references and uses a central shared counter to keep all counts on track. This is ideal because each core updates its counts by modifying its per-core counter, usually only needing access to its own local cache, cutting down on waiting for locks or serialization. Sloppy counters are also backwards-compatible with existing shared-counter code, making its implementation much easier to accomplish. The main disadvantages of the sloppy counters are that in situations where object de-allocation occurs often, because the de-allocation itself is an expensive operation, and the counters use up space proportional to the number of cores.

====Lock-free comparison: ''Section 4.4''====
This section describes a specific instance of unnecessary locking.

====Per-Core Data Structures: ''Section 4.5''====
Three centralized data structures were causing bottlenecks - a per-superblock list of open files, vfsmount table, the packet buffers free list. Each data structure was decentralized to per-core versions of itself. In the case of vfsmount the central data structure was maintained, and any per-core misses got written from the central table to the per-core table.

====Eliminating false sharing: ''Section 4.6''====
Misplaced variables on the cache cause different cores to request the same line to be read and written at the same time often enough to significantly impact performance. By moving the often written variable to another line the bottleneck was removed.

====Avoiding unnecessary locking: ''Section 4.7''====
Many locks/mutexes have special cases where they don't need to lock. Likewise mutexes can be split from locking the whole data structure to locking a part of it. Both these changes remove or reduce bottlenecks.

====Conclusion: ''Sections 6 & 7''====
Everything so far indicates that MOSBENCH application can scale to 48 cores. This scaling required a few modest changes to remove bottlenecks. The MIT team speculate that that trend will continue as the number of cores increase. They also state that things not bottlenecked by the CPU are harder to fix.

We can eliminate most kernel bottlenecks that the applications hits most often with minor changes. Most changes were well known methodology, with the exception of Sloppy counters. This study is limited by the removal of the IO bottleneck, but it does suggest that traditional implementations can be made scalable.

==Critique==
What is good and not-so-good about this paper? You may discuss both the style and content;
be sure to ground your discussion with specific references. Simple assertions that something is good or bad is not enough - you must explain why.

Since this is a "my implementation is better then your implementation" paper the "goodness" of content can be impartially determined by its fairness and the honesty of the authors.

====Content(Fairness): ''Section 5''====

=====memcached: ''Section 5.3''=====
memcached is treated with near perfect fairness in the paper. Its an in-memory service, so the ignored storage IO bottleneck does not affect it at all. Likewise the "stock" and "PK" implementations are given the same test suite, so there is no advantage given to either. memcached itself is non-scalable, so the MIT team was forced to run one instance per-core to keep up throughput. The FAQ at memcached.org's wiki suggests using multiple implementations per-server as a work around to another problem, which implies that running multiple instances of the server is the same, or nearly the same, as running one larger server [1]. In the end memcached was bottlenecked by the network card.

[1] memcached's wiki: http://code.google.com/p/memcached/wiki/FAQ#Can_I_use_different_size_caches_across_servers_and_will_memcache

=====Apache: ''Section 5.4''=====
Linux has a built in kernel flaw where network packets are forced to travel though multiple queues before they arrive at queue where they can be processed by the application. This imposes significant costs on multi-core systems due to queue locking costs. This flaw inherently diminishes the performance of Apache on multi-core system due to multiple threads spread across cores being forced to deal with these mutex (mutual exclusion) costs. For the sake of this experiment Apache had a separate instance on every core listening on different ports which is not a practical real world application but merely an attempt to implement better parallel execution on a traditional kernel. The patched kernel implementation of the network stack is also specific to the problem at hand, which is processing multiple short lived connections across multiple cores. Although this provides a performance increase in the given scenario, in more general applications network performance might suffer. These tests were also rigged to avoid bottlenecks in place by network and file storage hardware. Meaning, making the proposed modifications to the kernel wont necessarily produce the same increase in productivity as described in the article. This is very much evident in the test where performance degrades past 36 cores due to limitation of the networking hardware. ''Which is not a problem as the paper specifically states that there are hardware limitations.''

=====gmake: ''Section 5.6''=====
Since the inherent nature of gmake makes it quite parallel, the testing and updating that were attempted on gmake resulted in essentially the same scalability results for both the stock and modified kernel. The only change that was found was that gmake spent slightly less time at the system level because of the changes that were made to the system's caching. As stated in the paper, the execution time of gmake relies quite heavily on the compiler that is uses with gmake, so depending on which compiler was chosen, gmake could run worse or even slightly better. In any case, there seems to be no fairness concerns when it comes to the scalability testing of gmake as the same application load-out was used for all of the tests.

====Style====
Style Criterion (feel free to add I have no idea what should go here):
- does the paper present information out of order?
- does the paper present needless information?
- does the paper have any sections that are inherently confusing? Wrong? or use bad methodology?
- is the paper easy to read through, or does it change subjects repeatedly?
- does the paper have too many "long-winded" sentences, making it seem like the authors are just trying to add extra words to make it seem more important? - I think maybe limit this to run-on sentences.
- Check for grammar

===References===
[1] Silas Boyd-Wickizer et al. "An Analysis of Linux Scalability to Many Cores". In ''OSDI '10, 9th USENIX Symposium on OS Design and Implementation'', Vancouver, BC, Canada, 2010. http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf.

You will almost certainly have to refer to other resources; please cite these resources in the style of citation of the papers assigned (inlined numbered references). Place your bibliographic entries in this section.

gmake:

[http://www.gnu.org/software/make/manual/make.html gmake Manual]

[http://www.gnu.org/software/make/ gmake Main Page]

===Deprecated===
====Background Concepts====
* Exim: ''Section 3.1'':
**Exim is a mail server for Unix. It's fairly parallel. The server forks a new process for each connection and twice to deliver each message. It spends 69% of its time in the kernel on a single core.

* PostgreSQL: ''Section 3.4'':
**As implied by the name PostgreSQL is a SQL database. PostgreSQL starts a separate process for each connection and uses kernel locking interfaces extensively to provide concurrent access to the database. Due to bottlenecks introduced in its code and in the kernel code, the amount of time PostgreSQL spends in the kernel increases very rapidly with addition of new cores. On a single core system PostgreSQL spends only 1.5% of its time in the kernel. On a 48 core system the execution time in the kernel jumps to 82%.

* Psearchy: ''Section 3.6''

* Metis: ''Section 3.7''

Talk:COMP 3000 Essay 2 2010 Question 1

2010-12-02T15:53:47Z

Abown: /* Critique */

=Class and Notices=
(Nov. 30, 2010) Prof. Anil stated that we should focus on the 3 easiest to understand parts in section 5 and elaborate on them.

- Also, I, Daniel B., work Thursday night, so I will be finishing up as much of my part as I can for the essay before Thursday morning's class, then maybe we can all meet up in a lab in HP and put the finishing touches on the essay. I will be available online Wednesday night from about 6:30pm onwards and will be in the game dev lab or CCSS lounge Wednesday morning from about 11am to 2pm if anyone would like to meet up with me at those times.

- [[I suggest we meet up Thursday morning after Operating Systems in order to discuss and finalize the essay. Maybe we can even designate a lab for the group to meet up in. Any suggestions?]] - Daniel B.

- HP 3115 since there wont be a class in there (as its our tutorial and we know there won't be anyone there)
-- Go to Wireless Lab next to CCSS Lounge. Andrew and Dan B. will be there.

- If its all the same to you guys mind if I just join you via msn or iirc? Or phone if you really want. -Rannath

- I'm working today, but I'll be at a computer reading this page/contributing to my section. Depending on how busy I am, I should be able to get some significant writing in before 4pm today on my section and any additional sections required. RP

- I wont be there either. that does not mean i wont/cant contribute. I'll be on msn or you can just email me. -kirill

=Group members=
Patrick Young [mailto:Rannath@gmail.com Rannath@gmail.com]

Daniel Beimers [mailto:demongyro@gmail.com demongyro@gmail.com]

Andrew Bown [mailto:abown2@connect.carleton.ca abown2@connect.carleton.ca]

Kirill Kashigin [mailto:k.kashigin@gmail.com k.kashigin@gmail.com]

Rovic Perdon [mailto:rperdon@gmail.com rperdon@gmail.com]

Daniel Sont [mailto:dan.sont@gmail.com dan.sont@gmail.com]

=Methodology=
We should probably have our work verified by at least one group member before posting to the actual page

=To Do=
*Improve the grammar/structure of the paper section & add links to supplementary info
*Background Concepts -fill in info (fii)
*Contribution -fii
*Critique -fii
*References -fii

===Claim Sections===
* I claim Exim and memcached for background and critique -[[Rannath]]
* also per-core data structures, false sharing and unessesary locking for contribution -[[Rannath]]
* For starters I will take the Scalability Tutorial and gmake. Since the part for gmake is short in the paper, I will grab a few more sections later on. - [[Daniel B.]]
* Also, I will take sloppy counters as well - [[Daniel B.]]
* I'm gonna put some work into the apache and postgresql sections - kirill
* Just as a note Anil in class Thuesday the 30th of November said that we only need to explain 3 of the applications and not all 7 - [[Andrew]]
* I'll do the Research problem and contribution sections. - [[Andrew]]
* I will work on contribution - [[Rovic]]
* I'm gonna whip something up for 4.2 since there appears to be nothing mentioned about it. -kirill

* So here is the claims and unclaimed section. Add your name next to one if you want to take it on.
** gmake - Daniel B.
** memcached - Rannath
** Apache - Kirill
** [[(Exim, PostgreSQL, Metis, and Psearchy will not be needed as the professor said we only need to explain 3)]]
** Research Problem - Andrew
** Contribution - Rovic
** Critic, Style - Everyone
** Conclusion (also discussion) - Rannath, but I need someone to help flesh it out, I got the salient points down.
** References - Everyone

=Essay=
===Paper===
This paper was authored by - Silas Boyd-Wickizer, Austin T. Clements, Yandong Mao, Aleksey Pesterev, M. Frans Kaashoek, Robert Morris, and Nickolai Zeldovich.

They all work at MIT CSAIL.

[http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf The paper: An Analysis of Linux Scalability to Many Cores]

===Background Concepts===
Explain briefly the background concepts and ideas that your fellow classmates will need to know first in order to understand your assigned paper.

Ideas to explain:
- thread (maybe)
- Linux's move towards scalability precedes this paper. (assert this, no explanation needed, maybe a few examples)
- Summarize scalability tutorial (Section 4.1 of the paper) focus on what makes something (non-)scalable
- Describe the programs tested (what they do, how they're programmed (serial vs parallel), where to the do their processing)

====memchached: ''Section 3.2''====
memcached is an in-memory hash table server. One instance running on many cores is bottlenecked by an internal lock. The MIT team ran one instance per-core to avoid the problem. Clients each connect to a single instance. This allows the server to simulate parallelism. With few requests, memcached spends 80% of its time in the kernel on one core, mostly processing packets.

====Apache: ''Section 3.3''====
Apache is a web server. In the case of this study, Apache has been configured to run a separate process on each core. Each process, in turn, has multiple threads (Making it a perfect example of parallel programming). One thread to service incoming connections and various other threads to service those connections. On a single core processor, Apache spends 60% of its execution time in the kernel.

====gmake: ''Section 3.5''====
gmake is an unofficial default benchmark in the Linux community which is used in this paper to build the Linux kernel. gmake is already quite parallel, creating more processes than cores, so that it can make proper use of multiple cores, and involves much reading and writing of files, as it is used to build the Linux kernel. gmake is limited in scalability due to the serial processes that run at the beginning and end of its execution. gmake spends much of its execution time with its compiler, but still spend 7.6% of its time in system time. [1]

===Research problem===
my references are just below because it is easier for numbering the data later.

As technological progress the number of core a main processor can have is increasing at an impressive rate. Soon personal computers will have so many cores that scalability will be an issue. There has to be a way that standard user level Linux kernel will scale with a 48-core system[1]. The problem with a standard Linux operating is they are not designed for massive scalability which will soon be a problem. The issue with scalability is that a solo core will perform much more work compared to a single core working with 47 other cores. Although traditional logic that situation makes sense because 48 cores are dividing the work. But when processing information a process the main goal is to finish so as long as possible every core should be doing a much work as possible.

To fix those scalability issues it is necessary to focus on three major areas: the Linux kernel, user level design and how application use of kernel services. The Linux kernel can be improved be to improve sharing and have the advantage of recent iterations are beginning to implement scalability features. On the user level design applications can be improved so that there is more focus on parallelism since some programs have not implements those improved features. The final aspect of improving scalability is how an application uses kernel services to share resources better so that different aspects of the program are not conflicting over the same services. All of the bottlenecks are found actually only take a little work to avoid.[1]

This research is based on much research which was created before in the development of scalability for UNIX system. The major developments from shared memory machines [2], wait-free synchronization to fast message passing have created a base set of techniques which can be used to improve scalability. These techniques have been incorporated in all major operation system including Linux, Mac OS X and Windows. Linux has been improved with kernel subsystems such as Read-Copy-Update which an algorithm for which is used to avoid locks and atomic instructions which lower scalability.[3] The is also an excellent base a research on Linux scalability studies to base this research paper. These paper include a on doing scalability on a 32-core machine. [4] That research can improve the results by learning from the experiments already performed by researchers. This research also aid identifying bottlenecks which speed up researching solutions for those bottlenecks.

[2] J. Kuskin, D. Ofelt, M. Heinrich, J. Heinlein, R. Simoni, K. Gharachorloo, J. Chapin, D. Nakahira, J. Baxter, M. Horowitz, A. Gupta, M. Rosenblum, and J. Hennessy. The Stanford FLASH multiprocessor. In Proc. of the 21st ISCA, pages 302–313,1994.

[3] P. E. McKenney, D. Sarma, A. Arcangeli, A. Kleen, O. Krieger, and R. Russell. Read-copy-update. In Proceedings of the Linux Symposium 2002, pages 338-367, Ottawa Ontario, June 2002

[4] C. Yan, Y. Chen, and S. Yuanchun. OSMark: A benchmark suite for understanding parallel scalability of operating systems on large scale multi-cores. In 2009 2nd International Conference on Computer Science and Information Technology, pages 313–317, 2009

==Section 4.1 problems:==
**The percentage of serialization in a program has a lot to do with how much an application can be sped up. As from the example in the paper, it seems to follow Amdahl's law (e.g. 25% serialization --> limit of 4x speedup).
**Types of serializing interactions found in the MOSBENCH apps:
***Locking of shared data structure - increasing # of cores --> increase in lock wait time
***Writing to shared memory - increasing # of cores --> increase in wait for cache coherence protocol
***Competing for space in shared hardware cache - increasing # of cores --> increase in cache miss rate
***Competing for shared hardware resources - increasing # of cores --> increase in wait for resources
***Not enough tasks for cores --> idle cores

===Contribution===
What was implemented? Why is it any better than what came before?

Summarize info from Section 4.2 onwards, maybe put graphs from Section 5 here to provide support for improvements (if that isn't unethical/illegal)?

- So long as we cite the paper and don't pretend the graphs are ours, we are ok, since we are writing an explanation/critic of the paper.

-I'm just using this as a notepad, do not copy/paste this section, I will put in a properly written set of paragraphs which will fit with the contribution questions asked. -RP

-==Work in Progress==-- -Rovic P.
This research contributes by evaluating the scalability discrepancies of applications programming and kernel programming. Key discoveries in this research show the effectiveness of the kernel in handling scaling amongst CPU cores. This has also shown that scaling in application programming should be more the focus. It has been shown that simple scaling techniques (list techniques) such as programming parallelism (look up more stuff to back this up and quotes). (Sloppy counter effectiveness, possible positive contributions, what has been used (internet search), what hasn’t been used.) Read conclusion, 2nd paragraph.

One reason the
required changes are modest is that stock Linux already
incorporates many modifications to improve scalability.
More speculatively, perhaps it is the case that Linux’s
system-call API is well suited to an implementation that
avoids unnecessary contention over kernel objects.

====Multicore packet processing: ''Section 4.2''====

====Sloppy counters: ''Section 4.3''====
Bottlenecks were encountered when the applications undergoing testing were referencing and updating shared counters for multiple cores. The solution in the paper is to use sloppy counters, which gets each core to track its own separate counts of references and uses a central shared counter to keep all counts on track. This is ideal because each core updates its counts by modifying its per-core counter, usually only needing access to its own local cache, cutting down on waiting for locks or serialization. Sloppy counters are also backwards-compatible with existing shared-counter code, making its implementation much easier to accomplish. The main disadvantages of the sloppy counters are that in situations where object de-allocation occurs often, because the de-allocation itself is an expensive operation, and the counters use up space proportional to the number of cores.

====Lock-free comparison: ''Section 4.4''====
This section describes a specific instance of unnecessary locking.

====Per-Core Data Structures: ''Section 4.5''====
Three centralized data structures were causing bottlenecks - a per-superblock list of open files, vfsmount table, the packet buffers free list. Each data structure was decentralized to per-core versions of itself. In the case of vfsmount the central data structure was maintained, and any per-core misses got written from the central table to the per-core table.

====Eliminating false sharing: ''Section 4.6''====
Misplaced variables on the cache cause different cores to request the same line to be read and written at the same time often enough to significantly impact performance. By moving the often written variable to another line the bottleneck was removed.

====Avoiding unnecessary locking: ''Section 4.7''====
Many locks/mutexes have special cases where they don't need to lock. Likewise mutexes can be split from locking the whole data structure to locking a part of it. Both these changes remove or reduce bottlenecks.

====Conclusion: ''Sections 6 & 7''====
Everything so far indicates that MOSBENCH application can scale to 48 cores. This scaling required a few modest changes to remove bottlenecks. The MIT team speculate that that trend will continue as the number of cores increase. They also state that things not bottlenecked by the CPU are harder to fix.

We can eliminate most kernel bottlenecks that the applications hits most often with minor changes. Most changes were well known methodology, with the exception of Sloppy counters. This study is limited by the removal of the IO bottleneck, but it does suggest that traditional implementations can be made scalable.

==Critique==
What is good and not-so-good about this paper? You may discuss both the style and content;
be sure to ground your discussion with specific references. Simple assertions that something is good or bad is not enough - you must explain why.

Since this is a "my implementation is better then your implementation" paper the "goodness" of content can be impartially determined by its fairness and the honesty of the authors.

====Content(Fairness): ''Section 5''====

=====memcached: ''Section 5.3''=====
memcached is treated with near perfect fairness in the paper. Its an in-memory service, so the ignored storage IO bottleneck does not affect it at all. Likewise the "stock" and "PK" implementations are given the same test suite, so there is no advantage given to either. memcached itself is non-scalable, so the MIT team was forced to run one instance per-core to keep up throughput. The FAQ at memcached.org's wiki suggests using multiple implementations per-server as a work around to another problem, which implies that running multiple instances of the server is the same, or nearly the same, as running one larger server [1]. In the end memcached was bottlenecked by the network card.

[1] memcached's wiki: http://code.google.com/p/memcached/wiki/FAQ#Can_I_use_different_size_caches_across_servers_and_will_memcache

=====Apache: ''Section 5.4''=====
Linux has a built in kernel flaw where network packets are forced to travel though multiple queues before they arrive at queue where they can be processed by the application. This imposes significant costs on multi-core systems due to queue locking costs. This flaw inherently diminishes the performance of Apache on multi-core system due to multiple threads spread across cores being forced to deal with these mutex (mutual exclusion) costs. For the sake of this experiment Apache had a separate instance on every core listening on different ports which is not a practical real world application but merely an attempt to implement better parallel execution on a traditional kernel. The patched kernel implementation of the network stack is also specific to the problem at hand, which is processing multiple short lived connections across multiple cores. Although this provides a performance increase in the given scenario, in more general applications network performance might suffer. These tests were also rigged to avoid bottlenecks in place by network and file storage hardware. Meaning, making the proposed modifications to the kernel wont necessarily produce the same increase in productivity as described in the article. This is very much evident in the test where performance degrades past 36 cores due to limitation of the networking hardware. ''Which is not a problem as the paper specifically states that there are hardware limitations.''

=====gmake: ''Section 5.6''=====
Since the inherent nature of gmake makes it quite parallel, the testing and updating that were attempted on gmake resulted in essentially the same scalability results for both the stock and modified kernel. The only change that was found was that gmake spent slightly less time at the system level because of the changes that were made to the system's caching. As stated in the paper, the execution time of gmake relies quite heavily on the compiler that is uses with gmake, so depending on which compiler was chosen, gmake could run worse or even slightly better. In any case, there seems to be no fairness concerns when it comes to the scalability testing of gmake as the same application load-out was used for all of the tests.

====Style====
Style Criterion (feel free to add I have no idea what should go here):
- does the paper present information out of order?
- does the paper present needless information?
- does the paper have any sections that are inherently confusing? Wrong? or use bad methodology?
- is the paper easy to read through, or does it change subjects repeatedly?
- does the paper have too many "long-winded" sentences, making it seem like the authors are just trying to add extra words to make it seem more important? - I think maybe limit this to run-on sentences.
- Check for grammar

===References===
[1] Silas Boyd-Wickizer et al. "An Analysis of Linux Scalability to Many Cores". In ''OSDI '10, 9th USENIX Symposium on OS Design and Implementation'', Vancouver, BC, Canada, 2010. http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf.

You will almost certainly have to refer to other resources; please cite these resources in the style of citation of the papers assigned (inlined numbered references). Place your bibliographic entries in this section.

gmake:

[http://www.gnu.org/software/make/manual/make.html gmake Manual]

[http://www.gnu.org/software/make/ gmake Main Page]

===Deprecated===
====Background Concepts====
* Exim: ''Section 3.1'':
**Exim is a mail server for Unix. It's fairly parallel. The server forks a new process for each connection and twice to deliver each message. It spends 69% of its time in the kernel on a single core.

* PostgreSQL: ''Section 3.4'':
**As implied by the name PostgreSQL is a SQL database. PostgreSQL starts a separate process for each connection and uses kernel locking interfaces extensively to provide concurrent access to the database. Due to bottlenecks introduced in its code and in the kernel code, the amount of time PostgreSQL spends in the kernel increases very rapidly with addition of new cores. On a single core system PostgreSQL spends only 1.5% of its time in the kernel. On a 48 core system the execution time in the kernel jumps to 82%.

* Psearchy: ''Section 3.6''

* Metis: ''Section 3.7''

Talk:COMP 3000 Essay 2 2010 Question 1

2010-12-02T05:39:55Z

Abown: /* Research problem */

=Class and Notices=
(Nov. 30, 2010) Prof. Anil stated that we should focus on the 3 easiest to understand parts in section 5 and elaborate on them.

- Also, I, Daniel B., work Thursday night, so I will be finishing up as much of my part as I can for the essay before Thursday morning's class, then maybe we can all meet up in a lab in HP and put the finishing touches on the essay. I will be available online Wednesday night from about 6:30pm onwards and will be in the game dev lab or CCSS lounge Wednesday morning from about 11am to 2pm if anyone would like to meet up with me at those times.

- [[I suggest we meet up Thursday morning after Operating Systems in order to discuss and finalize the essay. Maybe we can even designate a lab for the group to meet up in. Any suggestions?]] - Daniel B.

- HP 3115 since there wont be a class in there (as its our tutorial and we know there won't be anyone there)

- If its all the same to you guys mind if I just join you via msn or iirc? Or phone if you really want.

=Group members=
Patrick Young [mailto:Rannath@gmail.com Rannath@gmail.com]

Daniel Beimers [mailto:demongyro@gmail.com demongyro@gmail.com]

Andrew Bown [mailto:abown2@connect.carleton.ca abown2@connect.carleton.ca]

Kirill Kashigin [mailto:k.kashigin@gmail.com k.kashigin@gmail.com]

Rovic Perdon [mailto:rperdon@gmail.com rperdon@gmail.com]

Daniel Sont [mailto:dan.sont@gmail.com dan.sont@gmail.com]

=Methodology=
We should probably have our work verified by at least one group member before posting to the actual page

=To Do=
*Improve the grammar/structure of the paper section & add links to supplementary info
*Background Concepts -fill in info (fii)
*Research problem -fii
*Contribution -fii
*Critique -fii
*References -fii

===Claim Sections===
* I claim Exim and memcached for background and critique -[[Rannath]]
* also per-core data structures, false sharing and unessesary locking for contribution -[[Rannath]]
* For starters I will take the Scalability Tutorial and gmake. Since the part for gmake is short in the paper, I will grab a few more sections later on. - [[Daniel B.]]
* Also, I will take sloppy counters as well - [[Daniel B.]]
* I'm gonna put some work into the apache and postgresql sections - kirill
* Just as a note Anil in class Thuesday the 30th of November said that we only need to explain 3 of the applications and not all 7 - [[Andrew]]
* I'll do the Research problem and contribution sections. - [[Andrew]]
* I will work on contribution - [[Rovic]]

=Essay=
===Paper===
This paper was authored by - Silas Boyd-Wickizer, Austin T. Clements, Yandong Mao, Aleksey Pesterev, M. Frans Kaashoek, Robert Morris, and Nickolai Zeldovich.

They all work at MIT CSAIL.

[http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf The paper: An Analysis of Linux Scalability to Many Cores]

===Background Concepts===
Explain briefly the background concepts and ideas that your fellow classmates will need to know first in order to understand your assigned paper.

Ideas to explain:
- thread (maybe)
- Linux's move towards scalability precedes this paper. (assert this, no explanation needed, maybe a few examples)
- Summarize scalability tutorial (Section 4.1 of the paper) focus on what makes something (non-)scalable
- Describe the programs tested (what they do, how they're programmed (serial vs parallel), where to the do their processing)

====Exim: ''Section 3.1''====
Exim is a mail server for Unix. It's fairly parallel. The server forks a new process for each connection and twice to deliver each message. It spends 69% of its time in the kernel on a single core.

====memchached: ''Section 3.2''====
memcached is an in-memory hash table. memchached is very much not parallel, but can be made to be, just run multiple instances. Have clients worry about synchronizing data between the different instances. With few requests memcached does most of its processing at the network stack, 80% of its time on one core.

====Apache: ''Section 3.3''====
Apache is a web server. In the case of this study, Apache has been configured to run a separate process on each core. Each process, in turn, has multiple threads (Making it a perfect example of parallel programming). One thread to service incoming connections and various other threads to service those connections. On a single core processor, Apache spends 60% of its execution time in the kernel.

====PostgreSQL: ''Section 3.4''====
As implied by the name PostgreSQL is a SQL database. PostgreSQL starts a separate process for each connection and uses kernel locking interfaces extensively to provide concurrent access to the database. Due to bottlenecks introduced in its code and in the kernel code, the amount of time PostgreSQL spends in the kernel increases very rapidly with addition of new cores. On a single core system PostgreSQL spends only 1.5% of its time in the kernel. On a 48 core system the execution time in the kernel jumps to 82%.

====gmake: ''Section 3.5''====
gmake is an unofficial default benchmark in the Linux community which is used in this paper to build the Linux kernel. gmake is already quite parallel, creating more processes than cores, so that it can make proper use of multiple cores, and involves much reading and writing of files, as it is used to build the Linux kernel. gmake is limited in scalability due to the serial processes that run at the beginning and end of its execution. gmake spends much of its execution time with its compiler, but still spend 7.6% of its time in system time.

====Psearchy: ''Section 3.6''====

====Metis: ''Section 3.7''====

===Research problem===
my references are just below because it is easier for numbering the data later.

As technological progress the number of core a main processor can have is increasing at an impressive rate. Soon personal computers will have so many cores that scalability will be an issue. There has to be a way that standard user level Linux kernel will scale with a 48-core system[1]. The problem with a standard Linux operating is they are not designed for massive scalability which will soon be a problem. The issue with scalability is that a solo core will perform much more work compared to a single core working with 47 other cores. Although traditional logic that situation makes sense because 48 cores are dividing the work. But when processing information a process the main goal is to finish so as long as possible every core should be doing a much work as possible.

To fix those scalability issues it is necessary to focus on three major areas: the Linux kernel, user level design and how application use of kernel services. The Linux kernel can be improved be to improve sharing and have the advantage of recent iterations are beginning to implement scalability features. On the user level design applications can be improved so that there is more focus on parallelism since some programs have not implements those improved features. The final aspect of improving scalability is how an application uses kernel services to share resources better so that different aspects of the program are not conflicting over the same services. All of the bottlenecks are found actually only take a little work to avoid.[1]

This research is based on much research which was created before in the development of scalability for UNIX system. The major developments from shared memory machines [2], wait-free synchronization to fast message passing have created a base set of techniques which can be used to improve scalability. These techniques have been incorporated in all major operation system including Linux, Mac OS X and Windows. Linux has been improved with kernel subsystems such as Read-Copy-Update which an algorithm for which is used to avoid locks and atomic instructions which lower scalability.[3] The is also an excellent base a research on Linux scalability studies to base this research paper. These paper include a on doing scalability on a 32-core machine. [4] That research can improve the results by learning from the experiments already performed by researchers. This research also aid identifying bottlenecks which speed up researching solutions for those bottlenecks.

[2] J. Kuskin, D. Ofelt, M. Heinrich, J. Heinlein, R. Simoni, K. Gharachorloo, J. Chapin, D. Nakahira, J. Baxter, M. Horowitz, A. Gupta, M. Rosenblum, and J. Hennessy. The Stanford FLASH multiprocessor. In Proc. of the 21st ISCA, pages 302–313,1994.

[3] P. E. McKenney, D. Sarma, A. Arcangeli, A. Kleen, O. Krieger, and R. Russell. Read-copy-update. In Proceedings of the Linux Symposium 2002, pages 338-367, Ottawa Ontario, June 2002

[4] C. Yan, Y. Chen, and S. Yuanchun. OSMark: A benchmark suite for understanding parallel scalability of operating systems on large scale multi-cores. In 2009 2nd International Conference on Computer Science and Information Technology, pages 313–317, 2009

==Section 4.1 problems:==
**The percentage of serialization in a program has a lot to do with how much an application can be sped up. As from the example in the paper, it seems to be an inverse relationship (e.g. 25% serialization --> limit of 4x speedup).
**Types of serializing interactions found in the MOSBENCH apps:
***Locking of shared data structure - increasing # of cores --> increase in lock wait time
***Writing to shared memory - increasing # of cores --> increase in wait for cache coherence protocol
***Competing for space in shared hardware cache - increasing # of cores --> increase in cache miss rate
***Competing for shared hardware resources - increasing # of cores --> increase in wait for resources
***Not enough tasks for cores --> idle cores

===Contribution===
What was implemented? Why is it any better than what came before?

Summarize info from Section 4.2 onwards, maybe put graphs from Section 5 here to provide support for improvements (if that isn't unethical/illegal)?
- So long as we cite the paper and don't pretend the graphs are ours, we are ok, since we are writing an explanation/critic of the paper.

====Multicore packet processing: ''Section 4.2''====

====Sloppy counters: ''Section 4.3''====
Bottlenecks were encountered when the applications undergoing testing were referencing and updating shared counters for multiple cores. The solution in the paper is to use sloppy counters, which gets each core to track its own separate counts of references and uses a central shared counter to keep all counts on track. This is ideal because each core updates its counts by modifying its per-core counter, usually only needing access to its own local cache, cutting down on waiting for locks or serialization. Sloppy counters are also backwards-compatible with existing shared-counter code, making its implementation much easier to accomplish. The main disadvantages of the sloppy counters are that in situations where object de-allocation occurs often, because the de-allocation itself is an expensive operation, and the counters use up space proportional to the number of cores.

====Lock-free comparison: ''Section 4.4''====

====Per-Core Data Structures: ''Section 4.5''====
Three centralized data structures were causing bottlenecks - a per-superblock list of open files, vfsmount table, the packet buffers free list. Each data structure was decentralized to per-core versions of itself. In the case of vfsmount the central data structure was maintained, and any per-core misses got written from the central table to the per-core table.

====Eliminating false sharing: ''Section 4.6''====
Misplaced variables on the cache cause different cores to request the same line to be read and written at the same time often enough to significantly impact performance. By moving the often written variable to another line the bottleneck was removed.

====Avoiding unnecessary locking: ''Section 4.7''====
Many locks/mutexes have special cases where they don't need to lock. Likewise mutexes can be split from locking the whole data structure to locking a part of it. Both these changes remove or reduce bottlenecks.

====Conclusion====
Conclusion: we can make a traditional OS architecture scale (at least to 48 cores), we just have to remove bottlenecks.

===Critique===
What is good and not-so-good about this paper? You may discuss both the style and content;
be sure to ground your discussion with specific references. Simple assertions that something is good or bad is not enough - you must explain why.

Since this is a "my implementation is better then your implementation" paper the "goodness" of content can be impartially determined by its fairness and the honesty of the authors.

====Content(Fairness): ''Section 5''====
Fairness criterion:
- does the test accurately describe real-world use-cases (or some set there-of)? (external fairness, maybe ignored for testing and benchmarking purposes, usually is too)
- does the test put all tested implementations through the same test? (internal fairness)

Both the stock and new implementations use the same benchmarks, therefore neither of them has a particular advantage. That holds true for all seven programs. In spite of this, there are also some assumptions or conditions that the paper fails to provide a fair explanation as to the inclusion/exclusion of.

=====Exim: ''Section 5.2''=====
The test uses a relatively small number of connections, but that is also implicitly stated to be a non-issue - "as long as there are enough clients to keep Exim busy, the number of clients has little effect on performance."

This test is explicitly stated to be ignoring the real-world constraint of the IO bottleneck, thus is unfair when compared to real-world scenarios. The purpose was not to test the IO bottleneck. Therefore the unfairness to real-world scenarios is unimportant.

=====memcached: ''Section 5.3''=====
memcached has no explicit or implicit fairness concerns with respect to real-world scenarios.

=====Apache: ''Section 5.4''=====
Linux has a built in kernel flaw where network packets are forced to travel though multiple queues before they arrive at queue where they can be processed by the application. This imposes significant costs on multi-core systems due to queue locking costs. This flaw inherently diminishes the performance of Apache on multi-core system due to multiple threads spread across cores being forced to deal with these mutex (mutual exclusion) costs. For the sake of this experiment Apache had a separate instance on every core listening on different ports which is not a practical real world application but merely an attempt to implement better parallel execution on a traditional kernel. These tests were also rigged to avoid bottlenecks in place by network and file storage hardware. Meaning, making the proposed modifications to the kernel wont necessarily produce the same increase in productivity as described in the article. This is very much evident in the test where performance degrades past 36 cores due to limitation of the networking hardware.

=====PostgreSQL: ''Section 5.5''=====

=====gmake: ''Section 5.6''=====
Since the inherent nature of gmake makes it quite parallel, the testing and updating that were attempted on gmake resulted in essentially the same scalability results for both the stock and modified kernel. The only change that was found was that gmake spent slightly less time at the system level because of the changes that were made to the system's caching. As stated in the paper, the execution time of gmake relies quite heavily on the compiler that is uses with gmake, so depending on which compiler was chosen, gmake could run worse or even slightly better. In any case, there seems to be no fairness concerns when it comes to the scalability testing of gmake as the same application load-out was used for all of the tests.

=====Psearchy: ''Section 5.7''=====

=====Metis: ''Section 5.8''=====

====Style====
Style Criterion (feel free to add I have no idea what should go here):
- does the paper present information out of order?
- does the paper present needless information?
- does the paper have any sections that are inherently confusing?
- is the paper easy to read through, or does it change subjects repeatedly?
- does the paper have too many "long-winded" sentences, making it seem like the authors are just trying to add extra words to make it seem more important? - I think maybe limit this to run-on sentences.
- Check for grammar

===References===
You will almost certainly have to refer to other resources; please cite these resources in the style of citation of the papers assigned (inlined numbered references). Place your bibliographic entries in this section.

gmake:

[http://www.gnu.org/software/make/manual/make.html gmake Manual]

[http://www.gnu.org/software/make/ gmake Main Page]

Talk:COMP 3000 Essay 2 2010 Question 1

2010-12-02T01:43:39Z

Abown: /* Research problem */

Talk:COMP 3000 Essay 2 2010 Question 1

2010-12-01T23:52:11Z

Abown: /* To Do */

Talk:COMP 3000 Essay 2 2010 Question 1

2010-12-01T23:50:19Z

Abown: /* Claim Sections */

Talk:COMP 3000 Essay 2 2010 Question 1

2010-11-22T17:26:03Z

Abown: /* Group members */

=Group members=
Patrick Young [mailto:Rannath@gmail.com Rannath@gmail.com]

Daniel Beimers [mailto:demongyro@gmail.com demongyro@gmail.com]

Andrew Bown [mailto:abown2@connect.carleton.ca abown2@connect.carleton.ca]

=Methodology=
We should probably have our work verified by at least one group member before posting to the actual page

=To Do=
*Improve the grammar/structure of the paper section
*Background Concepts -fill in info (fii)
*Research problem -fii
*Contribution -fii
*Critique -fii
*References -fii

===Claim Sections===
* I claim Exim and memcached for background and critique -[[Rannath]]
* also per-core data structures, false sharing and unessesary locking for contribution -[[Rannath]]

=Essay=
===Paper===
The paper's title, authors, and their affiliations. Include a link to the paper and any particularly helpful supplementary information.

Authors in order presented: Silas Boyd-Wickizer, Austin T. Clements, Yandong Mao, Aleksey Pesterev, M. Frans Kaashoek, Robert Morris, and Nickolai Zeldovich

affiliation: MIT CSAIL

[http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf An Analysis of Linux Scalability to Many Cores]

===Background Concepts===
Explain briefly the background concepts and ideas that your fellow classmates will need to know first in order to understand your assigned paper.

Ideas to explain:
#thread (maybe)
#Linux's move towards scalability precedes this paper. (assert this, no explanation needed, maybe a few examples)
#Summarize scalability tutorial (Section 4.1 of the paper)
#Describe the programs tested (what they do, how they're programmed (serial vs parallel), where to the do their processing)

=====Exim: ''Section 3.1''=====
Exim is a mail server for Unix. It's fairly parallel. The server forks a new process for each connection and twice to deliver each message. It spends 69% of its time in the kernel on a single core.

=====memchached: ''Section 3.2''=====
memcached is an in-memory hash table. memchached is very much not parallel, but can be made to be, just run multiple instances. Have clients worry about synchronizing data between the different instances. With few requests memcached does most of its processing at the network stack, 80% of its time on one core.

===Research problem===
What is the research problem being addressed by the paper? How does this problem relate to past related work?

Problem being addressed: scalability of current generation OS architecture, using Linux as an example. (?)

Summarize related works (Section 2, include links, expand information to have at least a summary of some related work)

===Contribution===
What was implemented? Why is it any better than what came before?

Summarize info from Section 4.2 onwards, maybe put graphs from Section 5 here to provide support for improvements (if that isn't unethical/illegal)?

Conclusion: we can make a traditional OS architecture scale (at least to 48 cores), we just have to remove bottlenecks.

=====Per-Core Data Structures=====
Three centralized data structures were causing bottlenecks - a per-superblock list of open files, vfsmount table, the packet buffers free list. Each data structure was decentralized to per-core versions of itself. In the case of vfsmount the central data structure was maintained, and any per-core misses got written from the central table to the per-core table.

=====Eliminating false sharing=====
Misplaced variables on the cache cause different cores to request the same line to be read and written at the same time often enough to significantly impact performance. By moving the often written variable to another line the bottleneck was removed.

=====Avoiding unnecessary locking=====
Many locks/mutexes have special cases where they don't need to lock. Likewise mutexes can be split from locking the whole data structure to locking a part of it. Both these changes remove or reduce bottlenecks.

===Critique===
What is good and not-so-good about this paper? You may discuss both the style and content; be sure to ground your discussion with specific references. Simple assertions that something is good or bad is not enough - you must explain why.

Since this is a "my implementation is better then your implementation" paper the "goodness" of content can be impartially determined by its fairness and the honesty of the authors.

Fairness criterion:
#does the test accurately describe real-world use-cases (or some set there-of)? (external fairness, maybe ignored for testing and benchmarking purposes, usually is too)
#does the test put all tested implementations through the same test? (internal fairness)

Style Criterion (feel free to add I have no idea what should go here):
#does the paper present information out of order?
#does the paper present needless information?
#does the paper have any sections that are inherently confusing?

=====Testing Method: ''Section 5''=====
Both the stock and new implementations use the same benchmarks, therefore internal fairness is preserved for all seven programs.

=====Exim: ''Section 5.2''=====
The test uses a relatively small number of connections, but that is also implicitly stated to be a non-issue - "as long as there are enough clients to keep Exim busy, the number of clients has little effect on performance."

This test is explicitly stated to be ignoring the real-world constraint of the IO bottleneck, thus is unfair when compared to real-world scenarios. The purpose was not to test the IO bottleneck. Therefore the unfairness to real-world scenarios is unimportant.

=====memcached: ''Section 5.3''=====
memcached has no explicit or implicit fairness concerns with respect to real-world scenarios.

===References===
You will almost certainly have to refer to other resources; please cite these resources in the style of citation of the papers assigned (inlined numbered references). Place your bibliographic entries in this section.

COMP 3000 Essay 2 2010 Question 8

2010-11-16T13:44:51Z

Abown: added skeleton

=Paper=
The paper's title, authors, and their affiliations. Include a link to the paper and any particularly helpful supplementary information.

=Background Concepts=
Explain briefly the background concepts and ideas that your fellow classmates will need to know first in order to understand your assigned paper.

=Research problem=
What is the research problem being addressed by the paper? How does this problem relate to past related work?

=Contribution=
What are the research contribution(s) of this work? Specifically, what are the key research results, and what do they mean? (What was implemented? Why is it any better than what came before?)

=Critique=
What is good and not-so-good about this paper? You may discuss both the style and content; be sure to ground your discussion with specific references. Simple assertions that something is good or bad is not enough - you must explain why.

=References=
You will almost certainly have to refer to other resources; please cite these resources in the style of citation of the papers assigned (inlined numbered references). Place your bibliographic entries in this section.

COMP 3000 Essay 1 2010 Question 3

2010-11-09T14:33:37Z

Abown: added one weakness of windows clusters

=Question=

To what extent do modern Windows systems provide mainframe-equivalent functionality? What about Windows coupled with add-on commercial products such as VMWare's virtualization and EMC's storage solutions? Explain.

=Answer=

'''This essay does a good job of discussing how Windows provides mainframe-class capabilities; it does not do a good job of addressing where it falls short. --Anil'''

Throughout the years, Windows has undergone some rather radical changes by modernizing existing technologies and providing innovation to existing features; this resulted in having functionality equivalent to that of a mainframe computer. However, although these changes have been extensive, Windows has not been particularly dominant when it comes to replacing modern mainframe systems.

== Mainframes ==

Mainframe systems have always had a good reputation for being used by large organizations in order to process thousands of small transactions. Whether these systems are used by the bank or by a police department, they possess several key features which make them exceedingly more powerful when compared to other systems. One of these features is an extensive and prolonged stability. This is a result of having tremendous redundancy and exception handling which prevents the entire system from shutting down, even if some components are inactive due to unforeseen circumstances. Because of this, mainframe computers are incredibly reliable when it comes to data storage and interoperability.

With this in mind, another neat feature that a mainframe possesses is the ability to hot swap components without taking the system offline. Consequently, components that are malfunctioning or require an upgrade can safely be replaced without endangering system stability. As a result mainframes gain a broad life spectrum as components can be upgraded individually without having to replace the entire system. Additionally, software written for these machines is extremely backwards compatible. The reason behind this is the fact that mainframe computers are fully virtualized. This is what allows a mainframe to run software that could have been written decades ago while still being able to run alongside modern software and hardware. In addition, this is part of the reason why mainframe computers are so secure, it is because they can use a combination of newer and older software as well as hardware to take years of innovation and combine it into one secure platform.

Unfortunately, all these features would mean nothing if the mainframe could not keep up with the data being sent and received. As a result, computers of this calibre must be able to have good I/O resource management as well as protect against bottlenecks. They do this by supporting powerful schedulers which ensure the fastest possible throughput for transaction processing [http://www.exforsys.com/tutorials/mainframe/mainframe-features.html]. Without this, you could continuously be upgrading components but suffer diminishing returns.

With so many features, how is Windows expected to keep up? The reality is Windows already supports most of these features. And when coupled with addon software such as VMWare and EMC storage solutions, the capabilities are even more astounding.

==Redundancy ==

A large feature of mainframes is their ability for redundancy. Mainframes produce redundancy by using the provider's off-site redundancy faeture. This feature lets the customer move all of their processes and applications onto the providers mainframe while the provider makes repairs on the customers system. Another way that mainframes create redundancy is their use of multi-processors that share the same memory. If one processor dies, the rest of the processors still keep all of the cache.

There are multiple ways windows systems can create this redundancy feature that mainframes have. The first way windows systems can create this is by creating a windows cluster server. The cluster uses the same feature of the mainframe's multi-processor system.This service allows multiple Windows machines to be connected as nodes in a cluster; where each node has the same applications and only one node is online at any point in time. If a node in the cluster fails, another will take over. The failing node can then be restarted or replaced without serious downtime. However this service does not offer fault tolerance to the same extent as actual mainframes. Another way windows systems can create redundancy is by using virtual machines. VMWare has a feature called Microsoft Cluster Service, which allows users to create a cluster of virtual machines on one physical windows system (or multiple physical machines). The virtual machines set up two different networks. They create a private network for communication in between the virtual machines and then a public network to control I/O services. The virtual machines also share storage to create concurrency so that if one fails, the other still has all of the data.But if the failure is on the Windows host machine then they will all fail. The virtual cluster can be maintained across multiple machines, allowing multiple users to have the reliability of clusters on fewer machines.

== No downtime upgrades ==

A useful feature that mainframes have is the ability to hot-swap. Hot-swapping is the ability to swap out components of a computer/mainframe for new components with no downtime (i.e. the system continues to run through this process). Hot-swapping occurs when there is faulty hardware in one of the processors inside the mainframe. Technicians are able to swap out this component without the mainframe being turned off or crashing. Hot-swapping is also used when upgrading processors, memory and storage inside the mainframe. With the right software and setup (redundancy) a mainframe is able to be upgraded and/or repaired as is sees fit by adding and removing components such as hard drives and processors.

Using VMWare on a Windows system allows users to hot-add RAM and hot-plug adds a new virtual CPU to the virtualized system. Using these hot-adding and hot-plugging techniques the virtual computer can grow in size to be able to accept loads varying in size. Due to some circumstances with different CPU's and guest OS's, the virtual machine may have to restart and is unable to hot-add/hot-plug. For example, the virtual machine of a Windows Server 2008 ENT x64 OS allows you to hot-add memory but you must restart it to remove memory and to add/remove CPU.

In non-virtual systems, Windows coupled with the program Go-HotSwap can hot-plug CompactPCI components. CompactPCI components allow many different devices to be plugged into their slots (e.g. multiple SATA hard drives) which makes a Windows system with these technologies very modular. The Windows Server 2008 R2 Datacenter released in 2009 uses dynamic hardware partitioning. Dynamic hardware partitioning means that its hardware can be partitioned into separate hardware with processors and other components which allows for hot-swapping/hot-adding of these partitions where needed.

== Backwards-Compatibility ==
Backwards-compatibility means that the newer software version can recognize what the old version write and how it work. It is a relationship between the two versions. If the new components provide all the functionality of the old one, we said that the new component is backwards compatible.In computer mainframe era, many applications are backwards compatible.For example,the code written 20 years ago in IBM System/360 can be run in latest mainframe (like zSeries, System/390 family,System z9,etc).This because that models in mainframe computer provide a combination of special hardware,special microcode and an emulation program to simulate the target system.(The IBM 7080 transistorized computer was backward compatible with all models of the IBM 705 vacuum tube computer.)

In Windows OS,one method to implement backwards-compatibility is to add applications. Like Microsoft Windows Application Compatibility Toolkit.This application can make the platform to be compatible with most softwares from early version.The second method is the Windows Operating Systems usually have various subsystems.The software originally designed for older version or other OSs can be run in the subsystems.Such as Window NT, it has MS-DOS and Win16 subsystems. Virtualization (running an older version of the OS on top of the new one) can also be used to enable old applications to run. (Windows 7 can do this with its virtualized XP subsystem.) The third method is to use shims to create the backwards-compatibility. Shims are just like the small libraries that can intercept the API, change parameters passed,handle and redirect the operations. In Windows OS,we can use shims to simulate the behaviors of old version OS for legacy softwares.

== I/O and Resource Management ==
Throughput, unlike input and output, is the measurement of the number of calculations per second that a machine can preform. This is usually measured in FLOPS (floating point logical operations per second). It is impossible for one sole Windows machine to compete with a mainframe's throughput. Not only do mainframe processors have extremely high frequencies, but they also have a considerable amount of cores. This all changes, however, when computer clustering is introduced. In the recent years, IBM has constructed a clustered system called The Roadrunner that ranks third in the TOP500 supercomputer list as of June 2010.[http://hubpages.com/hub/Most-Powerful-Computers-In-The-World] It has a total of 60 connected units, over a thousand processors, and the capability of computing at a rate of 1.7 petaflops.

The question is, with such complex hardware, how is it possible for any sort of software to use this clustered system? Luckily, Windows has introduced an OS called Windows Compute Cluster Server, which provides the necessary software to allow the main computer to utilize the computing power of its cluster nodes. Windows mainly uses MS-MPI (Microsoft Message Passing Interface) to send messages via Ethernet to its other nodes.[http://webcache.googleusercontent.com/search?q=cache:EPlDExBxmDYJ:download.microsoft.com/download/9/e/d/9edcdeab-f1fb-4670-8914-c08c5c6f22a5/HPC_Overview.doc+Windows+Compute+Cluster+Server&cd=1&hl=en&ct=clnk&gl=ca&client=firefox-a] Developers can use this function because it automatically connects a given process to each node. Windows then can use its scheduler to determine which node receives each different job. It keeps track of each node, and shuts the job down once the output is received.

Today, clustering computers together with the intent of optimizing throughput is accomplished using grid computing. Grid computing shares the same basic idealisms of cluster computing, however, grids have the sole job of computing massive scale problems.[http://searchdatacenter.techtarget.com/definition/grid-computing] Each subsection of a problem is passed out to a compute node in the grid for it to be calculated. The one clear problem of this computational model is that the problem must have the ability to be broken down into several pieces for each compute node to work on. This style of high throughput computing can be used for problems such as high-energy physics, or biology models.

In general, however, the most popular solution to solve problems that require large throughput would be to construct a cluster model. Most businesses require the reliability of clusters, even though it sacrifices performance; there is no competition to the hight availability of a cluster server as compared to the grid model.[http://www.dba-oracle.com/real_application_clusters_rac_grid/grid_vs_clusters.htm]

== Weaknesses of Cluster Computing ==
Although are strong at duplicating duplicating mainframe capability it does have it's own weaknesses. One such weakness is that adding extra computers to a cluster may not cause the system to run faster. Depending on the type of process required to run there will be no increase in speed and in some cases a decrease in speed. When the a OSEM (Ordered subset expectation–maximization) algorithm or a Katsevich algorithm is run, the ideal number of computer in the cluster is in fact different. In the OSEM algorithm has a linear speed-up for each added node until the 10th node and then has a plateau of speed increase. Whereas the Katsevich algorithm has a linear speed increase until the 16th node added[http://www.cs.uiowa.edu/~jni/publications/2006a%20Tao%20He%20IMSCCS06.pdf]. So depending on the process the clusters can handle the operation better or worse.

== Conclusion ==
Windows has gone from a Operation System specialize for personal computers to a platform to create a replacement for a mainframe. It quickly stripped away all the advantages that a normal mainframe has. But the largest threat Windows has to Mainframes is the cost. It gives anyone the ability to create a mainframe equivalent system with stock parts comparatively cheap price. Does this mean that the mainframe time is running out? Only Time will tell.

== References ==

=== Redundancy ===
"Setup for Failover Clustering and Microsoft Cluster Service" <http://www.vmware.com/pdf/vsphere4/r40/vsp_40_mscs.pdf>

"Introducing Microsoft Cluster Service (MSCS) in the Windows Server 2003 Family" <http://msdn.microsoft.com/en-us/library/ms952401.aspx>

=== No downtime upgrades ===
Davis, David. "VMware vSphere hot-add RAM and hot-plug CPU." TechTarget. N.p., 09.15.2009. Web. 14 Oct 2010. <http://searchvmware.techtarget.com/tip/0,289483,sid179_gci1367631_mem1,00.html>.

"Windows Server 2008 R2 Datacenter." Windows Server 2008 R2. N.p., n.d. Web. 14 Oct 2010. <http://www.microsoft.com/windowsserver2008/en/us/2008-dc.aspx>.

"Go-HotSwap: CompactPCI Hot Swap." Jungo. Jungo Ltd, n.d. Web. 14 Oct 2010. <http://www.jungo.com/st/hotswap.html>.

=== Backwards-Compatibility ===

"Windows 7 To Break Backwards Compatibility" <http://www.windows7news.com/2008/05/23/windows-7-to-break-backwards-compatibility/>

"Mainframe computers" <http://computersight.com/computers/mainframe-computers/>

"Mainframe Features" <http://www.scribd.com/doc/6895677/Mainframe-Features>

"Mapping the Mainframe to Windows: A Reference Architecture" <http://www.microsoft.com/windowsserver/mainframe/papers.mspx>

=== I/O and Resource Management ===

"Most Powerful Computers In The World" <http://hubpages.com/hub/Most-Powerful-Computers-In-The-World>

Windows Server 2003 Administrator’s Companion (MS Press, 2003) "Overview of Microsoft Windows Compute Cluster Server 2003" <http://webcache.googleusercontent.com/search?q=cache:EPlDExBxmDYJ:download.microsoft.com/download/9/e/d/9edcdeab-f1fb-4670-8914-c08c5c6f22a5/HPC_Overview.doc+Windows+Compute+Cluster+Server&cd=1&hl=en&ct=clnk&gl=ca&client=firefox-a>

"grid computing" <http://searchdatacenter.techtarget.com/definition/grid-computing>

"Clusters vs. Grids" <http://www.dba-oracle.com/real_application_clusters_rac_grid/grid_vs_clusters.htm>

<http://publib.boulder.ibm.com/infocenter/tpfhelp/current/index.jsp?topic=/com.ibm.ztpf-ztpfdf.doc_put.cur/gtpc3/c3thru.html>

<http://searchcio-midmarket.techtarget.com/sDefinition/0,,sid183_gci213140,00.html>

=== Weaknesses of Cluster Computing ===
Tao He, Jun Ni and Ge Wang,"A Heterogeneous Windows Cluster System for Medical Image Reconstruction",Proceedings of the 35th annual ACM/IEEE international symposium on Microarchitecture, IEEE, 2008, http://www.cs.uiowa.edu/~jni/publications/2006a%20Tao%20He%20IMSCCS06.pdf, (2010-11-09)

COMP 3000 Lab 4 2010

2010-11-02T14:39:50Z

Abown: /* Pro-tip */

All of the following should be done with an Ubuntu 10.04 distribution or equivalent. We recommend experimenting in a virtual environment because some of the exercises could make your system unbootable. (In fact, take a snapshot of your working system before starting these exercises so you can easily revert.)

==Questions==

# Change the grub command line at boot to limit the total available RAM to 256M. You'll need to get to select an entry and edit it from within grub.
# Add a new grub menu item which limits the standard kernel to 256M.
# Add a second virtual disk and make it bootable: put the kernel and initial ram disk on it and then install grub. Can you boot off of this disk? What does it do?
# Examine the standard kernel's initial ram disk (initrd). What program is first run in this environment? What does it do?
# Modify the standard initial RAM disk so it pauses for 10 seconds and prints a message to the console on boot.
# What programs does upstart start on boot?

==Hints==

Please add your hints below to help your fellow students!

=== Pro-tip ===
sudo update-grub (important!!! guess what it does!!!)

===Kernel command line options===

===GRUB configuration===

*On Ubuntu the user configuration is stored in /etc/default/grub, while system grub configuration is in /etc/grub.d. The main grub files are stored in /boot/grub. You can update grub's config with the update-grub command.
*to limit RAM, go to /etc/default/grub, and change default boot options so that mem=256M is included.

===How GRUB works===

===Making a disk bootable===

===Examining RAM disks===

*Ubuntu (Debian) store initial RAM disks in the cpio format. 'zcat <file> | cpio -i' will extract its contents.

===Upstart/init===

Upstart "jobs" are config (.conf) files in /etc/init that require one of two options an "exec" line or a "script"

The exec line allows the upstart to just simply execute a script elsewhere, while script allows you to shell script in the upstart job.

A special upstart job is rc.conf which maintains the original runlevel init.d scripts. You will see that rc.conf simply executes all the /etc/init.d scripts.

See more on upstart jobs at http://upstart.ubuntu.com/getting-started.html

For those using the newer versions of Ubuntu and thus using Grub2
https://help.ubuntu.com/community/Grub2

COMP 3000 Lab 4 2010

2010-11-02T14:39:37Z

Abown: /* Hints */

All of the following should be done with an Ubuntu 10.04 distribution or equivalent. We recommend experimenting in a virtual environment because some of the exercises could make your system unbootable. (In fact, take a snapshot of your working system before starting these exercises so you can easily revert.)

==Questions==

# Change the grub command line at boot to limit the total available RAM to 256M. You'll need to get to select an entry and edit it from within grub.
# Add a new grub menu item which limits the standard kernel to 256M.
# Add a second virtual disk and make it bootable: put the kernel and initial ram disk on it and then install grub. Can you boot off of this disk? What does it do?
# Examine the standard kernel's initial ram disk (initrd). What program is first run in this environment? What does it do?
# Modify the standard initial RAM disk so it pauses for 10 seconds and prints a message to the console on boot.
# What programs does upstart start on boot?

==Hints==

Please add your hints below to help your fellow students!

=== Pro-tip ===
sudo update-grub (important!!! guess what it does!!!

===Kernel command line options===

===GRUB configuration===

*On Ubuntu the user configuration is stored in /etc/default/grub, while system grub configuration is in /etc/grub.d. The main grub files are stored in /boot/grub. You can update grub's config with the update-grub command.
*to limit RAM, go to /etc/default/grub, and change default boot options so that mem=256M is included.

===How GRUB works===

===Making a disk bootable===

===Examining RAM disks===

*Ubuntu (Debian) store initial RAM disks in the cpio format. 'zcat <file> | cpio -i' will extract its contents.

===Upstart/init===

Upstart "jobs" are config (.conf) files in /etc/init that require one of two options an "exec" line or a "script"

The exec line allows the upstart to just simply execute a script elsewhere, while script allows you to shell script in the upstart job.

A special upstart job is rc.conf which maintains the original runlevel init.d scripts. You will see that rc.conf simply executes all the /etc/init.d scripts.

See more on upstart jobs at http://upstart.ubuntu.com/getting-started.html

For those using the newer versions of Ubuntu and thus using Grub2
https://help.ubuntu.com/community/Grub2

Talk:COMP 3000 Essay 1 2010 Question 3

2010-10-15T05:19:33Z

Abown: /* Group 3 */

== Group 3 ==
Here's my email I'll add some of the stuff I find soon I'm just saving the question for last.
Andrew Bown(abown2@connect.carleton.ca)

I'm not sure if this is totally relevant, oh well.
-First time sharing system CTSS (Compatible Time Sharing System) in the 1950s. Created at MIT
http://www.kernelthread.com/publications/virtualization/

-achamney@connect.carleton.ca

Here's my contact info (qzhang13@connect.carleton.ca)
An article about the mainframe.
-Mainframe Migration http://www.microsoft.com/windowsserver/mainframe/migration.mspx

-[[User:Zhangqi|Zhangqi]] 15:02, 7 October 2010 (UTC)

Here's my contact information, look forward to working with everyone. - Ben Robson (brobson@connect.carleton.ca)

Hey, Here's my contact info, nshires@connect.carleton.ca, I'll have some sources posted by the weekend hopefully

Hey guys i'm not in your group but I found some useful information that could help you
http://en.wikipedia.org/wiki/Mainframe_computer i know we are not suppose to use wiki references but its a good place to start

Okay found an article paper titled called"Mainframe Scalability in the Windows Environment"
http://new.cmg.org/proceedings/2003/3023.pdf (required registration to access but is free)~ Andrew (abown2@connect.carleton.ca)sometime friday.

Folks, remember to do your discussions here. Use four tildes to sign your entries, that adds time and date. Email discussions won't count towards your participation grade...
[[User:Soma|Anil]] 15:43, 8 October 2010 (UTC)

Okay going to break the essay into points paragraphs on the main page which people can choose one paragraph to write. Then after all paragraphs are written we will communally edit it to have a cohesive voice. It is the only way I can viably think of to properly distribute the work. ~Andrew (abown2@connect.carleton.ca) 11:00 am, 10 October 2010.

Link to IBMs info on their mainframes --[[User:Lmundt|Lmundt]] 19:58, 7 October 2010 (UTC)
http://publib.boulder.ibm.com/infocenter/zos/basics/index.jsp?topic=/com.ibm.zos.zmainframe/zconc_valueofmf.htm

Just made the revelation that when trying to find information on the Windows equivalent to mainframe is refered to as '''clustering''' which should help finding information.
Here's the wiki article on the technology for an overview http://en.wikipedia.org/wiki/Microsoft_Cluster_Server ~ Andrew (abown2@connect.carleton.ca

hey,I agree with Andrew's idea. We should break the essay into several sections and work it together.From my point of view, I think we should focus on how Windows provide the mainframe functionality and the VMware and EMC's storage should be our examples. As listed on the main page, there are many advantages and disadvantages of the mainframe.But where is Windows? I'm confused...
In my opinion, the first paragraph can introduct the mainframe (such as the history,features,application,etc) and what mainframe-equivalent functionality Windows support. Then we can use some paragraphs to discuss the functionalities in details. And VMware and EMC's storage solution also can be involved in this part. At last we make a conclusion of the whloe essay. Do you think it's feasible?

--[[User:Zhangqi|Zhangqi]] 02:12, 11 October 2010 (UTC)

Ah but the question isn't the pros and cons of each. It is how to get mainframe functionality from a Windows Operating System. How I split up the essay has each paragraph focusing on one aspect of mainframes and how it can be duplicated in windows either with windows tools or 3rd party software. You don't need to go into the history or applications of mainframes since that is not required by the phrasing of the question.

~ Andrew Bown, 11:28 AM, October 11th 2010

Okay, I think I catch your meaning. So now we should do is to edit the content of each paragragh as soon as possible. Time is limited.

--[[User:Zhangqi|Zhangqi]] 19:57, 11 October 2010 (UTC)

If you guys are looking for an authoritative source on how Windows works, I *highly* recommend checking out "Window Internals 4th Edition" or "Windows Internals 5th Edition" by Mark Russinovich and David Solomon.

--[[User:3maisons|3maisons]] 18:59, 12 October 2010 (UTC)

Hey guys nice work, sorry I didn't have time to add more to the essay today. I combined the essay into a FrankenEssay which is on the front page and added a conclusion. If read through it but if anyone notices a mistake I missed go ahead and correct it.
--[[User:Abown|Andrew Bown]] 1:16, 15 October 2010

OLD VERSION - Here for the time being while optimizing some sections --[[User:Dkrutsko|Dkrutsko]] 00:20, 14 October 2010 (UTC)

=Answer=
added introduction points and sections for each paragraph so you guys can edit one paragraph at a time instead of the whole document. If you want to claim a certain paragram just put your name into the section first. ~ Andrew (abown2@connect.carleton.ca) 12:00 10th of October 2010

== Introduction ==
Main Aspects of mainframes:
* redundancy which enables high reliability and security
* high input/output
* backwards-compatibility with legacy software
* support massive throughput
* Systems run constantly so they can be hot upgraded
http://www.exforsys.com/tutorials/mainframe/mainframe-features.html

Linking sentence about how windows can duplicate mainframe functionality.

here's the introduction ~ Abown (11:12 pm, October 12th 2010) 
Thanks Abown, just tweaked a couple of the sentences to improve flow [[User:Achamney|Achamney]] 01:13, 14 October 2010 (UTC)

Also, i removed this statement "Unfortunately, computers are only able to process data as fast as they can receive it". I couldn't find a good place to plug it in.

Mainframes have been always used for large corporations to process thousands of small transactions, but what strengths allow for mainframes to be useful in their purpose. Mainframes are extremely useful in business because they are designed to run without downtime. This is achieved by having tremendous redundancy which allows for mainframes to be extremely reliable. This also gives security when concerning data loss due to downtime. Mainframes can be upgraded without taking the system down to allow for repairs, which further increases reliability. After upgrading a mainframe, however, the software does not change, so they can offer the features of backwards compatibility through virtualization; software never needs to be replaced. Mainframes support high input/output so that the mainframe is always being utilized. To make sure mainframes are utilized to their fullest, they support powerful schedulers which ensure the fastest throughput for processing transactions as fast as possible. [http://www.exforsys.com/tutorials/mainframe/mainframe-features.html] With so many features, how are Windows based systems supposed to compete with a mainframe? The fact of the matter is that there are features in Windows, and software solutions which can duplicate these features in a Windows environment. Be it redundancy, real-time upgrading, virtualization, high input/output or utilizing resources.

Using this paragraph and my solution on the assignment I was able to expand on this topic. It is in the main page at the moment, see if you like it, add anything you think I missed --[[User:Dkrutsko|Dkrutsko]] 05:17, 14 October 2010 (UTC)

== History ==
Before comparing Windows systems and mainframes, the history of what mainframes were used for and where they came from must be understood. The first official mainframe computer was the UNIVAC I. [http://www.vikingwaters.com/htmlpages/MFHistory.htm] It was designed for the U.S. Census Bureau by J. Presper Eckert and John Mauchly. [http://www.thocp.net/hardware/univac.htm]. By this point in history, there were no personal computers, and the only people who could afford a computer were massive businesses. The main functionality of these mainframes were to calculate company payrolls, sales records, analyze sales performance, and store all company information. 
[[User:Achamney|Achamney]] 01:30, 12 October 2010 (UTC)

This doesn't seem to actually be pertinent to the question at hand. Question does not have any indication of the need to provide a history. [[User:Abown|Andrew Bown]] 11:16, 12 October 2010

I have to agree this doesn't seem relevant to the question. --[[User:Dkrutsko|Dkrutsko]] 00:10, 14 October 2010 (UTC)

== Redundancy ==
[[User:Nshires|Nshires]] 04:10, 13 October 2010 (UTC)
A large feature of mainframes is their ability for redundancy. Mainframes produce redundancy by using the provider's off-site redundancy faeture. This feature lets the customer move all of their processes and applications onto the providers mainframe while the provider makes repairs on the customers system. Another way that mainframes create redundancy is their use of multi-processors that share the same memory. If one processor dies, the rest of the processors still keep all of the cache. There are multiple ways windows systems can create this redundancy feature that mainframes have. The first way windows systems can create this is by creating a windows cluster server. The cluster uses the same feature of the mainframe's multi-processor system. Another way windows systems can create redundancy is by using virtual machines. VMWare has a feature called Microsoft Cluster Service, which allows users to create a cluster of virtual machines on one physical windows system (or multiple physical machines). The virtual machines set up two different networks. They create a private network for communication in between the virtual machines and then a public network to control I/O services. The virtual machines also share storage to create concurrency so that if one fails, the other still has all of the data.

(this is what I've gotten out of some researching so far, comments and any edits/suggestions if I'm on the right track or not are greatly apreciated :) )
*note: This is the second time I have written this, make sure to save whatever you edit in notepad or whatever first so that you don't lose everything*

link to VMWare's cluster virtualization http://www.vmware.com/pdf/vsphere4/r40/vsp_40_mscs.pdf

[[User:Nshires|Nshires]] 04:10, 13 October 2010 (UTC)

:I'll attempt to re-write this paragraph for clarity and accuracy:

:A feature provided by mainframes is their ability to create redundancy in terms of data storage and parallel processing. Windows can mimic expandable storage and storage redundancy through out-sourced storage solutions.

:Processing redundancy for Windows can be created through the Microsoft Cluster Service (MSCS). This service allows multiple Windows machines to be connected as nodes in a cluster; where each node has the same applications and only one node is online at any point in time. If a node in the cluster fails, another will take over. The failing node can then be restarted or replaced without serious downtime. However this service does not offer fault tolerance to the same extent as actual mainframes.

:Source: http://msdn.microsoft.com/en-us/library/ms952401.aspx

:Virtual machine nodes can be used in place of physical machine nodes in a cluster, providing redundant application services to end-users. If the a virtual machine fails, other virtual machines can take over, if the failure is on the Windows host machine then they will all fail. The virtual cluster can be maintained across multiple machines, allowing multiple users to have the reliability of clusters on fewer machines.

:Let me know what you think.
:[[User:Brobson|Brobson]] 18:25, 14 October 2010 (UTC)

== hot swapping ==
[[User:Nshires|Nshires]] 16:47, 13 October 2010 (UTC)
Another useful feature that mainframes have is the ability to hot-swap. Hot-swapping occurs when there is faulty hardware in one of the processors inside the mainframe and technicians are able to swap out this component without the mainframe being turned off or crashing. Hot-swapping is also used when upgrading processors inside the mainframe. With the right software and setup (redundancy) a mainframe is able to upgrade and/or repair their mainframe as they see fit. Using VMWare on a Windows system allows users to hot-add RAM and hot-plug adds a new virtual CPU to the virtualized system. Using these hot-adding and hot-plugging techniques the virtual computer can grow in size to be able to accept loads varying in size. In non-virtual systems, Windows coupled with the program Go-HotSwap can hot-plug CompactPCI components. CompactPCI components allow many different devices to be plugged into their slots (e.g. multiple SATA hard drives) which makes a Windows system with these technologies very modular.

These are the concepts I've been able to figure out so far about hot-swapping/hot-upgrading, feel free to add/edit and what-not!

Sources:
http://searchvmware.techtarget.com/tip/0,289483,sid179_gci1367631,00.html
http://www.jungo.com/st/hotswap_windows.html
[[User:Nshires|Nshires]] 16:47, 13 October 2010 (UTC)

:According to your searchvmware.techtarget.com source, a processor cannot be hot-plugged in the truest sense of the word in that the hardware needs to be rebooted to recognize the added hardware. Hot-swapping demands zero downtime.
:If you don't mind me suggesting but I don't think this section should be referring to the hot-swapping/hot-adding/or hot-plugging of virtual machines or client machines of the mainframe. I think for hot-swapping we should focus on the hot-swapping of hardware components. As such we can point out that Windows does support mainframe-level hot-swapping with its Windows Server 2008 R2 Datacenter OS
:<blockquote>"Hot Add/Replace Memory and Processors with supporting hardware"</blockquote> http://www.microsoft.com/windowsserver2008/en/us/2008-dc.aspx

:If we are only consider the capabilities of the PC OS, then Windows only supports plug and play devices, such as external hard drives, and does not support RAM or CPU hot-swap.

:I'm also wondering if this should tie into scalability of a mainframe or if scalability should have it's own section.
:[[User:Brobson|Brobson]] 17:12, 14 October 2010 (UTC)

The source you mentioned talks about a virtual machine and that it can be hot-swapped with no downtime depending on the guest OS with no downtime. Some guest OS's need a reboot but some do not. The virtual Windows Server 2008 ENT x64 can hot-add memory with no downtime, it seems that no virtual os can hot-add cpu without rebooting. And the second part of my paragraph talks about physical Windows systems coupled with a program that enables hot-swapping of SATA hard drives and other components with no downtime.
I do agree that hot-swapping in a virtual machine may be kind of useless though haha :S. And I'll check out the Windows Server 2008 R2 Datacenter OS, Thanks [[User:Nshires|Nshires]] 00:33, 15 October 2010 (UTC)

Revised:
A useful feature that mainframes have is the ability to hot-swap. Hot-swapping is the ability to swap out components of a computer/mainframe for new components with no downtime (i.e. the system continues to run through this process). Hot-swapping occurs when there is faulty hardware in one of the processors inside the mainframe. Technicians are able to swap out this component without the mainframe being turned off or crashing. Hot-swapping is also used when upgrading processors, memory and storage inside the mainframe. With the right software and setup (redundancy) a mainframe is able to be upgraded and/or repaired as is sees fit by adding and removing components such as hard drives and processors.

Using VMWare on a Windows system allows users to hot-add RAM and hot-plug adds a new virtual CPU to the virtualized system. Using these hot-adding and hot-plugging techniques the virtual computer can grow in size to be able to accept loads varying in size. Due to some circumstances with different CPU's and guest OS's, the virtual machine may have to restart and is unable to hot-add/hot-plug. For example, the virtual machine of a Windows Server 2008 ENT x64 OS allows you to hot-add memory but you must restart it to remove memory and to add/remove CPU.

In non-virtual systems, Windows coupled with the program Go-HotSwap can hot-plug CompactPCI components. CompactPCI components allow many different devices to be plugged into their slots (e.g. multiple SATA hard drives) which makes a Windows system with these technologies very modular. The Windows Server 2008 R2 Datacenter released in 2009 uses dynamic hardware partitioning. Dynamic hardware partitioning means that its hardware can be partitioned into separate hardware with processors and other components which allows for hot-swapping/hot-adding of these partitions where needed.

Davis, David. "VMware vSphere hot-add RAM and hot-plug CPU." TechTarget. N.p., 09.15.2009. Web. 14 Oct 2010. <http://searchvmware.techtarget.com/tip/0,289483,sid179_gci1367631_mem1,00.html>.

"Windows Server 2008 R2 Datacenter." Windows Server 2008 R2. N.p., n.d. Web. 14 Oct 2010. <http://www.microsoft.com/windowsserver2008/en/us/2008-dc.aspx>.

"Go-HotSwap: CompactPCI Hot Swap." Jungo. Jungo Ltd, n.d. Web. 14 Oct 2010. <http://www.jungo.com/st/hotswap.html>.

feel free to edit [[User:Nshires|Nshires]] 03:49, 15 October 2010 (UTC)

== backwards-compatibility ==
Backwards-compatibility means that the newer software version can recognize what the old version write and how it work. It is a relationship between the two versions. If the new components provide all the functionality of the old one, we said that the new component is backwards compatible.In computer mainframe era, many applications are backwards compatible.For example,the code written 20 years ago in IBM System/360 can be run in latest mainframe (like zSeries, System/390 family,System z9,etc).This because that models in mainframe computer provide a combination of special hardware,special microcode and an emulation program to simulate the target system.(The IBM 7080 transistorized computer was backward compatible with all models of the IBM 705 vacuum tube computer.) Sometimes mainframe also need customers to halt the computer and download the emulation program.

In Windows OS,one method to implement backwards-compatibility is to add applications.Like Microsoft Windows Application Compatibility Toolkit.This application can make the platfrom to be compatible with most softwares from early version.The second method is the Windows Operating Systems usually have various subsystems.The software originally designed for older version or other OSs can be run in the subsystems.Such as Window NT, it has MS-DOS and Win16 subsystems.But Windows 7's backwards-compatibility is not very good.If kernel is different, the OSs can't be compatible with each other.But it doesn't mean that older programs won't run, virtualization will be used to make them run.The third method is to use shims to create the backwards-compatibility.Shims are just like the small libraries that can intercept the API, change parameters passed,handle and redirect the operations. In Windows OS,we can use shims to simulate the behaviors of old version OS for legacy softwares.

--[[User:Zhangqi|Zhangqi]] 08:34, 13 October 2010 (UTC)

ps. I didn't find perfect resources,just these.If you guys think any opinion is not correct,plz edit it or give suggestions :)

http://www.windows7news.com/2008/05/23/windows-7-to-break-backwards-compatibility/

http://computersight.com/computers/mainframe-computers/

Hey, this sounds really good, I'd add an example where you say 'one method to implement backward-compatibility is to add applications'.
And I did a little research and I found another way to create backwards compatibility using shims: http://en.wikipedia.org/wiki/Shim_%28computing%29
it pretty much intercepts the calls and changes them so that the old program can run on a new system.
Good Work, [[User:Nshires|Nshires]] 16:56, 13 October 2010 (UTC)

Thanks for your suggetions.I have added some information to the paragraph.:)
--[[User:Zhangqi|Zhangqi]] 00:24, 14 October 2010 (UTC)

== High input/output ==
~Andrew Bown (October 13 2:08) I'll write this paragraph.
I don't have time to write this before work(12-5) but I can put out the information i got already with research so if someone could help me complete this that it would be awesome since I have to finish up my 3004 document as well tonight.
~[User:Abown|Andrew Bown] (October 14th 11:12am)
Mainframes are able to achieve high/input output rates with their specialized Message Passing Interfaces (MPIs) which allow for fast intercommunication by sharing memory in between the different cores.https://www.mpitech.com/mpitech.nsf/pages/mainframe-&-AS400-printing_en.html

The latest versions of Windows clusters support a Microsoft created MPI surprisingly called Microsoft MPI[http://msdn.microsoft.com/en-us/library/bb524831(VS.85).aspx].

Microsoft's MPI is based off the MPICH2 explanation here:http://www.springerlink.com/content/hc4nyva6dvg6vdpp/

'''Looking at the details of the Microsoft MPI only runs if a process is put into the Microsoft Job Scheduler. So we may want to combine input/ouput and throughtput.'''

Hey guys.According to the resources above, the method for Windows to provide high input/output and massive throughput is almost the same. But I have no idea how to combine the two sections. Do we need to write somthing about input/output or just consider it under the massive throughput? [[User:Zhangqi|Zhangqi]] 22:38, 14 October 2010 (UTC)

== Massive Throughput ==
[[User:Achamney|Achamney]] 01:09, 14 October 2010 (UTC) 

[[User:Achamney|Achamney]] 21:18, 14 October 2010 (UTC) Done for now, I will come back to this after i get back (after 10:00pm tonight ish) and fix up the flow and such

Throughput, unlike input and output, is the measurement of the number of calculations per second that a machine can preform. This is usually measured in FLOPS (floating point logical operations per second). It is impossible for one sole Windows machine to compete with a mainframe's throughput. Not only do mainframe processors have extremely high frequencies, but they also have a considerable amount of cores. This all changes, however, when computer clustering is introduced. In the recent years, IBM has constructed a clustered system called The Roadrunner that ranks third in the TOP500 supercomputer list as of June 2010.[http://hubpages.com/hub/Most-Powerful-Computers-In-The-World] It has a total of 60 connected units, over a thousand processors, and the capability of computing at a rate of 1.7 petaflops.

The question is, with such complex hardware, how is it possible for any sort of software to use this clustered system? Luckily, Windows has introduced an OS called Windows Compute Cluster Server, which provides the necessary software to allow the main computer to utilize the computing power of its cluster nodes. Windows mainly uses MS-MPI (Microsoft Message Passing Interface) to send messages via Ethernet to its other nodes.[http://webcache.googleusercontent.com/search?q=cache:EPlDExBxmDYJ:download.microsoft.com/download/9/e/d/9edcdeab-f1fb-4670-8914-c08c5c6f22a5/HPC_Overview.doc+Windows+Compute+Cluster+Server&cd=1&hl=en&ct=clnk&gl=ca&client=firefox-a] Developers can use this function because it automatically connects a given process to each node. Windows then can use its scheduler to determine which node receives each different job. It keeps track of each node, and shuts the job down once the output is received.

Today, clustering computers together with the intent of optimizing throughput is accomplished using grid computing. Grid computing shares the same basic idealisms of cluster computing, however, grids have the sole job of computing massive scale problems.[http://searchdatacenter.techtarget.com/definition/grid-computing] Each subsection of a problem is passed out to a compute node in the grid for it to be calculated. The one clear problem of this computational model is that the problem must have the ability to be broken down into several pieces for each compute node to work on. This style of high throughput computing can be used for problems such as high-energy physics, or biology models.

In general, however, the most popular solution to solve problems that require large throughput would be to construct a cluster model. Most businesses require the reliability of clusters, even though it sacrifices performance; there is no competition to the hight availability of a cluster server as compared to the grid model.[http://www.dba-oracle.com/real_application_clusters_rac_grid/grid_vs_clusters.htm]

[http://publib.boulder.ibm.com/infocenter/tpfhelp/current/index.jsp?topic=/com.ibm.ztpf-ztpfdf.doc_put.cur/gtpc3/c3thru.html]
[http://searchcio-midmarket.techtarget.com/sDefinition/0,,sid183_gci213140,00.html]

COMP 3000 Essay 1 2010 Question 3

2010-10-15T05:14:58Z

Abown: /* Conclusion */

=Question=

To what extent do modern Windows systems provide mainframe-equivalent functionality? What about Windows coupled with add-on commercial products such as VMWare's virtualization and EMC's storage solutions? Explain.

=Answer=

Throughout the years, Windows has undergone some rather radical changes by modernizing existing technologies and providing innovation to existing features; this resulted in having functionality equivalent to that of a mainframe computer. However, although these changes have been extensive, Windows has not been particularly dominant when it comes to replacing modern mainframe systems.

== Mainframes ==

Mainframe systems have always had a good reputation for being used by large organizations in order to process thousands of small transactions. Whether these systems are used by the bank or by a police department, they possess several key features which make them exceedingly more powerful when compared to other systems. One of these features is an extensive and prolonged stability. This is a result of having tremendous redundancy and exception handling which prevents the entire system from shutting down, even if some components are inactive due to unforeseen circumstances. Because of this, mainframe computers are incredibly reliable when it comes to data storage and interoperability.

With this in mind, another neat feature that a mainframe possesses is the ability to hot swap components without taking the system offline. Consequently, components that are malfunctioning or require an upgrade can safely be replaced without endangering system stability. As a result mainframes gain a broad life spectrum as components can be upgraded individually without having to replace the entire system. Additionally, software written for these machines is extremely backwards compatible. The reason behind this is the fact that mainframe computers are fully virtualized. This is what allows a mainframe to run software that could have been written decades ago while still being able to run alongside modern software and hardware. In addition, this is part of the reason why mainframe computers are so secure, it is because they can use a combination of newer and older software as well as hardware to take years of innovation and combine it into one secure platform.

Unfortunately, all these features would mean nothing if the mainframe could not keep up with the data being sent and received. As a result, computers of this calibre must be able to have good I/O resource management as well as protect against bottlenecks. They do this by supporting powerful schedulers which ensure the fastest possible throughput for transaction processing [http://www.exforsys.com/tutorials/mainframe/mainframe-features.html]. Without this, you could continuously be upgrading components but suffer diminishing returns.

With so many features, how is Windows expected to keep up? The reality is Windows already supports most of these features. And when coupled with addon software such as VMWare and EMC storage solutions, the capabilities are even more astounding.

===Redundancy ===
A large feature of mainframes is their ability for redundancy. Mainframes produce redundancy by using the provider's off-site redundancy faeture. This feature lets the customer move all of their processes and applications onto the providers mainframe while the provider makes repairs on the customers system. Another way that mainframes create redundancy is their use of multi-processors that share the same memory. If one processor dies, the rest of the processors still keep all of the cache. There are multiple ways windows systems can create this redundancy feature that mainframes have. The first way windows systems can create this is by creating a windows cluster server. The cluster uses the same feature of the mainframe's multi-processor system. Another way windows systems can create redundancy is by using virtual machines. VMWare has a feature called Microsoft Cluster Service, which allows users to create a cluster of virtual machines on one physical windows system (or multiple physical machines). The virtual machines set up two different networks. They create a private network for communication in between the virtual machines and then a public network to control I/O services. The virtual machines also share storage to create concurrency so that if one fails, the other still has all of the data.
=== no downtime upgrades ===

A useful feature that mainframes have is the ability to hot-swap. Hot-swapping is the ability to swap out components of a computer/mainframe for new components with no downtime (i.e. the system continues to run through this process). Hot-swapping occurs when there is faulty hardware in one of the processors inside the mainframe. Technicians are able to swap out this component without the mainframe being turned off or crashing. Hot-swapping is also used when upgrading processors, memory and storage inside the mainframe. With the right software and setup (redundancy) a mainframe is able to be upgraded and/or repaired as is sees fit by adding and removing components such as hard drives and processors.

Using VMWare on a Windows system allows users to hot-add RAM and hot-plug adds a new virtual CPU to the virtualized system. Using these hot-adding and hot-plugging techniques the virtual computer can grow in size to be able to accept loads varying in size. Due to some circumstances with different CPU's and guest OS's, the virtual machine may have to restart and is unable to hot-add/hot-plug. For example, the virtual machine of a Windows Server 2008 ENT x64 OS allows you to hot-add memory but you must restart it to remove memory and to add/remove CPU.

In non-virtual systems, Windows coupled with the program Go-HotSwap can hot-plug CompactPCI components. CompactPCI components allow many different devices to be plugged into their slots (e.g. multiple SATA hard drives) which makes a Windows system with these technologies very modular. The Windows Server 2008 R2 Datacenter released in 2009 uses dynamic hardware partitioning. Dynamic hardware partitioning means that its hardware can be partitioned into separate hardware with processors and other components which allows for hot-swapping/hot-adding of these partitions where needed.

=== Backwards-Compatibility ===
Backwards-compatibility means that the newer software version can recognize what the old version write and how it work. It is a relationship between the two versions. If the new components provide all the functionality of the old one, we said that the new component is backwards compatible.In computer mainframe era, many applications are backwards compatible.For example,the code written 20 years ago in IBM System/360 can be run in latest mainframe (like zSeries, System/390 family,System z9,etc).This because that models in mainframe computer provide a combination of special hardware,special microcode and an emulation program to simulate the target system.(The IBM 7080 transistorized computer was backward compatible with all models of the IBM 705 vacuum tube computer.) Sometimes mainframe also need customers to halt the computer and download the emulation program.
In Windows OS,one method to implement backwards-compatibility is to add applications.Like Microsoft Windows Application Compatibility Toolkit.This application can make the platfrom to be compatible with most softwares from early version.The second method is the Windows Operating Systems usually have various subsystems.The software originally designed for older version or other OSs can be run in the subsystems.Such as Window NT, it has MS-DOS and Win16 subsystems.But Windows 7's backwards-compatibility is not very good.If kernel is different, the OSs can't be compatible with each other.But it doesn't mean that older programs won't run, virtualization will be used to make them run.The third method is to use shims to create the backwards-compatibility.Shims are just like the small libraries that can intercept the API, change parameters passed,handle and redirect the operations. In Windows OS,we can use shims to simulate the behaviors of old version OS for legacy softwares.

=== I/O and Resource Management ===
Throughput, unlike input and output, is the measurement of the number of calculations per second that a machine can preform. This is usually measured in FLOPS (floating point logical operations per second). It is impossible for one sole Windows machine to compete with a mainframe's throughput. Not only do mainframe processors have extremely high frequencies, but they also have a considerable amount of cores. This all changes, however, when computer clustering is introduced. In the recent years, IBM has constructed a clustered system called The Roadrunner that ranks third in the TOP500 supercomputer list as of June 2010.[http://hubpages.com/hub/Most-Powerful-Computers-In-The-World] It has a total of 60 connected units, over a thousand processors, and the capability of computing at a rate of 1.7 petaflops.

The question is, with such complex hardware, how is it possible for any sort of software to use this clustered system? Luckily, Windows has introduced an OS called Windows Compute Cluster Server, which provides the necessary software to allow the main computer to utilize the computing power of its cluster nodes. Windows mainly uses MS-MPI (Microsoft Message Passing Interface) to send messages via Ethernet to its other nodes.[http://webcache.googleusercontent.com/search?q=cache:EPlDExBxmDYJ:download.microsoft.com/download/9/e/d/9edcdeab-f1fb-4670-8914-c08c5c6f22a5/HPC_Overview.doc+Windows+Compute+Cluster+Server&cd=1&hl=en&ct=clnk&gl=ca&client=firefox-a] Developers can use this function because it automatically connects a given process to each node. Windows then can use its scheduler to determine which node receives each different job. It keeps track of each node, and shuts the job down once the output is received.

Today, clustering computers together with the intent of optimizing throughput is accomplished using grid computing. Grid computing shares the same basic idealisms of cluster computing, however, grids have the sole job of computing massive scale problems.[http://searchdatacenter.techtarget.com/definition/grid-computing] Each subsection of a problem is passed out to a compute node in the grid for it to be calculated. The one clear problem of this computational model is that the problem must have the ability to be broken down into several pieces for each compute node to work on. This style of high throughput computing can be used for problems such as high-energy physics, or biology models.

In general, however, the most popular solution to solve problems that require large throughput would be to construct a cluster model. Most businesses require the reliability of clusters, even though it sacrifices performance; there is no competition to the hight availability of a cluster server as compared to the grid model.[http://www.dba-oracle.com/real_application_clusters_rac_grid/grid_vs_clusters.htm]

=== Conclusion ===
Windows has gone from a Operation System specialize for personal computers to a platform to create a replacement for a mainframe. It quickly stripped away all the advantages that a normal mainframe has. But the largest threat Windows has to Mainframes is the cost. It gives anyone the ability to create a mainframe equivalent system with stock parts comparatively cheap price. Does this mean that the mainframe time is running out? Only Time will tell.

== References ==

=== no downtime upgrades ===
Davis, David. "VMware vSphere hot-add RAM and hot-plug CPU." TechTarget. N.p., 09.15.2009. Web. 14 Oct 2010. <http://searchvmware.techtarget.com/tip/0,289483,sid179_gci1367631_mem1,00.html>.

"Windows Server 2008 R2 Datacenter." Windows Server 2008 R2. N.p., n.d. Web. 14 Oct 2010. <http://www.microsoft.com/windowsserver2008/en/us/2008-dc.aspx>.

"Go-HotSwap: CompactPCI Hot Swap." Jungo. Jungo Ltd, n.d. Web. 14 Oct 2010. <http://www.jungo.com/st/hotswap.html>.

=== Backwards-Compatibility ===

http://www.windows7news.com/2008/05/23/windows-7-to-break-backwards-compatibility/
http://computersight.com/computers/mainframe-computers/

COMP 3000 Essay 1 2010 Question 3

2010-10-15T05:06:03Z

Abown: /* no downtime upgrades */

=Question=

To what extent do modern Windows systems provide mainframe-equivalent functionality? What about Windows coupled with add-on commercial products such as VMWare's virtualization and EMC's storage solutions? Explain.

=Answer=

Throughout the years, Windows has undergone some rather radical changes by modernizing existing technologies and providing innovation to existing features; this resulted in having functionality equivalent to that of a mainframe computer. However, although these changes have been extensive, Windows has not been particularly dominant when it comes to replacing modern mainframe systems.

== Mainframes ==

Mainframe systems have always had a good reputation for being used by large organizations in order to process thousands of small transactions. Whether these systems are used by the bank or by a police department, they possess several key features which make them exceedingly more powerful when compared to other systems. One of these features is an extensive and prolonged stability. This is a result of having tremendous redundancy and exception handling which prevents the entire system from shutting down, even if some components are inactive due to unforeseen circumstances. Because of this, mainframe computers are incredibly reliable when it comes to data storage and interoperability.

With this in mind, another neat feature that a mainframe possesses is the ability to hot swap components without taking the system offline. Consequently, components that are malfunctioning or require an upgrade can safely be replaced without endangering system stability. As a result mainframes gain a broad life spectrum as components can be upgraded individually without having to replace the entire system. Additionally, software written for these machines is extremely backwards compatible. The reason behind this is the fact that mainframe computers are fully virtualized. This is what allows a mainframe to run software that could have been written decades ago while still being able to run alongside modern software and hardware. In addition, this is part of the reason why mainframe computers are so secure, it is because they can use a combination of newer and older software as well as hardware to take years of innovation and combine it into one secure platform.

Unfortunately, all these features would mean nothing if the mainframe could not keep up with the data being sent and received. As a result, computers of this calibre must be able to have good I/O resource management as well as protect against bottlenecks. They do this by supporting powerful schedulers which ensure the fastest possible throughput for transaction processing [http://www.exforsys.com/tutorials/mainframe/mainframe-features.html]. Without this, you could continuously be upgrading components but suffer diminishing returns.

With so many features, how is Windows expected to keep up? The reality is Windows already supports most of these features. And when coupled with addon software such as VMWare and EMC storage solutions, the capabilities are even more astounding.

===Redundancy ===
A large feature of mainframes is their ability for redundancy. Mainframes produce redundancy by using the provider's off-site redundancy faeture. This feature lets the customer move all of their processes and applications onto the providers mainframe while the provider makes repairs on the customers system. Another way that mainframes create redundancy is their use of multi-processors that share the same memory. If one processor dies, the rest of the processors still keep all of the cache. There are multiple ways windows systems can create this redundancy feature that mainframes have. The first way windows systems can create this is by creating a windows cluster server. The cluster uses the same feature of the mainframe's multi-processor system. Another way windows systems can create redundancy is by using virtual machines. VMWare has a feature called Microsoft Cluster Service, which allows users to create a cluster of virtual machines on one physical windows system (or multiple physical machines). The virtual machines set up two different networks. They create a private network for communication in between the virtual machines and then a public network to control I/O services. The virtual machines also share storage to create concurrency so that if one fails, the other still has all of the data.
=== no downtime upgrades ===

A useful feature that mainframes have is the ability to hot-swap. Hot-swapping is the ability to swap out components of a computer/mainframe for new components with no downtime (i.e. the system continues to run through this process). Hot-swapping occurs when there is faulty hardware in one of the processors inside the mainframe. Technicians are able to swap out this component without the mainframe being turned off or crashing. Hot-swapping is also used when upgrading processors, memory and storage inside the mainframe. With the right software and setup (redundancy) a mainframe is able to be upgraded and/or repaired as is sees fit by adding and removing components such as hard drives and processors.

Using VMWare on a Windows system allows users to hot-add RAM and hot-plug adds a new virtual CPU to the virtualized system. Using these hot-adding and hot-plugging techniques the virtual computer can grow in size to be able to accept loads varying in size. Due to some circumstances with different CPU's and guest OS's, the virtual machine may have to restart and is unable to hot-add/hot-plug. For example, the virtual machine of a Windows Server 2008 ENT x64 OS allows you to hot-add memory but you must restart it to remove memory and to add/remove CPU.

In non-virtual systems, Windows coupled with the program Go-HotSwap can hot-plug CompactPCI components. CompactPCI components allow many different devices to be plugged into their slots (e.g. multiple SATA hard drives) which makes a Windows system with these technologies very modular. The Windows Server 2008 R2 Datacenter released in 2009 uses dynamic hardware partitioning. Dynamic hardware partitioning means that its hardware can be partitioned into separate hardware with processors and other components which allows for hot-swapping/hot-adding of these partitions where needed.

=== Backwards-Compatibility ===
Backwards-compatibility means that the newer software version can recognize what the old version write and how it work. It is a relationship between the two versions. If the new components provide all the functionality of the old one, we said that the new component is backwards compatible.In computer mainframe era, many applications are backwards compatible.For example,the code written 20 years ago in IBM System/360 can be run in latest mainframe (like zSeries, System/390 family,System z9,etc).This because that models in mainframe computer provide a combination of special hardware,special microcode and an emulation program to simulate the target system.(The IBM 7080 transistorized computer was backward compatible with all models of the IBM 705 vacuum tube computer.) Sometimes mainframe also need customers to halt the computer and download the emulation program.
In Windows OS,one method to implement backwards-compatibility is to add applications.Like Microsoft Windows Application Compatibility Toolkit.This application can make the platfrom to be compatible with most softwares from early version.The second method is the Windows Operating Systems usually have various subsystems.The software originally designed for older version or other OSs can be run in the subsystems.Such as Window NT, it has MS-DOS and Win16 subsystems.But Windows 7's backwards-compatibility is not very good.If kernel is different, the OSs can't be compatible with each other.But it doesn't mean that older programs won't run, virtualization will be used to make them run.The third method is to use shims to create the backwards-compatibility.Shims are just like the small libraries that can intercept the API, change parameters passed,handle and redirect the operations. In Windows OS,we can use shims to simulate the behaviors of old version OS for legacy softwares.

=== I/O and Resource Management ===
Throughput, unlike input and output, is the measurement of the number of calculations per second that a machine can preform. This is usually measured in FLOPS (floating point logical operations per second). It is impossible for one sole Windows machine to compete with a mainframe's throughput. Not only do mainframe processors have extremely high frequencies, but they also have a considerable amount of cores. This all changes, however, when computer clustering is introduced. In the recent years, IBM has constructed a clustered system called The Roadrunner that ranks third in the TOP500 supercomputer list as of June 2010.[http://hubpages.com/hub/Most-Powerful-Computers-In-The-World] It has a total of 60 connected units, over a thousand processors, and the capability of computing at a rate of 1.7 petaflops.

The question is, with such complex hardware, how is it possible for any sort of software to use this clustered system? Luckily, Windows has introduced an OS called Windows Compute Cluster Server, which provides the necessary software to allow the main computer to utilize the computing power of its cluster nodes. Windows mainly uses MS-MPI (Microsoft Message Passing Interface) to send messages via Ethernet to its other nodes.[http://webcache.googleusercontent.com/search?q=cache:EPlDExBxmDYJ:download.microsoft.com/download/9/e/d/9edcdeab-f1fb-4670-8914-c08c5c6f22a5/HPC_Overview.doc+Windows+Compute+Cluster+Server&cd=1&hl=en&ct=clnk&gl=ca&client=firefox-a] Developers can use this function because it automatically connects a given process to each node. Windows then can use its scheduler to determine which node receives each different job. It keeps track of each node, and shuts the job down once the output is received.

Today, clustering computers together with the intent of optimizing throughput is accomplished using grid computing. Grid computing shares the same basic idealisms of cluster computing, however, grids have the sole job of computing massive scale problems.[http://searchdatacenter.techtarget.com/definition/grid-computing] Each subsection of a problem is passed out to a compute node in the grid for it to be calculated. The one clear problem of this computational model is that the problem must have the ability to be broken down into several pieces for each compute node to work on. This style of high throughput computing can be used for problems such as high-energy physics, or biology models.

In general, however, the most popular solution to solve problems that require large throughput would be to construct a cluster model. Most businesses require the reliability of clusters, even though it sacrifices performance; there is no competition to the hight availability of a cluster server as compared to the grid model.[http://www.dba-oracle.com/real_application_clusters_rac_grid/grid_vs_clusters.htm]

== Conclusion ==
Again I don't think a conclusion is necessary unless its like one sentence. --[[User:Dkrutsko|Dkrutsko]] 23:43, 14 October 2010 (UTC)

== References ==

=== no downtime upgrades ===
Davis, David. "VMware vSphere hot-add RAM and hot-plug CPU." TechTarget. N.p., 09.15.2009. Web. 14 Oct 2010. <http://searchvmware.techtarget.com/tip/0,289483,sid179_gci1367631_mem1,00.html>.

"Windows Server 2008 R2 Datacenter." Windows Server 2008 R2. N.p., n.d. Web. 14 Oct 2010. <http://www.microsoft.com/windowsserver2008/en/us/2008-dc.aspx>.

"Go-HotSwap: CompactPCI Hot Swap." Jungo. Jungo Ltd, n.d. Web. 14 Oct 2010. <http://www.jungo.com/st/hotswap.html>.

=== Backwards-Compatibility ===

http://www.windows7news.com/2008/05/23/windows-7-to-break-backwards-compatibility/
http://computersight.com/computers/mainframe-computers/

COMP 3000 Essay 1 2010 Question 3

2010-10-15T05:05:40Z

Abown:

=Question=

To what extent do modern Windows systems provide mainframe-equivalent functionality? What about Windows coupled with add-on commercial products such as VMWare's virtualization and EMC's storage solutions? Explain.

=Answer=

Throughout the years, Windows has undergone some rather radical changes by modernizing existing technologies and providing innovation to existing features; this resulted in having functionality equivalent to that of a mainframe computer. However, although these changes have been extensive, Windows has not been particularly dominant when it comes to replacing modern mainframe systems.

== Mainframes ==

Mainframe systems have always had a good reputation for being used by large organizations in order to process thousands of small transactions. Whether these systems are used by the bank or by a police department, they possess several key features which make them exceedingly more powerful when compared to other systems. One of these features is an extensive and prolonged stability. This is a result of having tremendous redundancy and exception handling which prevents the entire system from shutting down, even if some components are inactive due to unforeseen circumstances. Because of this, mainframe computers are incredibly reliable when it comes to data storage and interoperability.

With this in mind, another neat feature that a mainframe possesses is the ability to hot swap components without taking the system offline. Consequently, components that are malfunctioning or require an upgrade can safely be replaced without endangering system stability. As a result mainframes gain a broad life spectrum as components can be upgraded individually without having to replace the entire system. Additionally, software written for these machines is extremely backwards compatible. The reason behind this is the fact that mainframe computers are fully virtualized. This is what allows a mainframe to run software that could have been written decades ago while still being able to run alongside modern software and hardware. In addition, this is part of the reason why mainframe computers are so secure, it is because they can use a combination of newer and older software as well as hardware to take years of innovation and combine it into one secure platform.

Unfortunately, all these features would mean nothing if the mainframe could not keep up with the data being sent and received. As a result, computers of this calibre must be able to have good I/O resource management as well as protect against bottlenecks. They do this by supporting powerful schedulers which ensure the fastest possible throughput for transaction processing [http://www.exforsys.com/tutorials/mainframe/mainframe-features.html]. Without this, you could continuously be upgrading components but suffer diminishing returns.

With so many features, how is Windows expected to keep up? The reality is Windows already supports most of these features. And when coupled with addon software such as VMWare and EMC storage solutions, the capabilities are even more astounding.

===Redundancy ===
A large feature of mainframes is their ability for redundancy. Mainframes produce redundancy by using the provider's off-site redundancy faeture. This feature lets the customer move all of their processes and applications onto the providers mainframe while the provider makes repairs on the customers system. Another way that mainframes create redundancy is their use of multi-processors that share the same memory. If one processor dies, the rest of the processors still keep all of the cache. There are multiple ways windows systems can create this redundancy feature that mainframes have. The first way windows systems can create this is by creating a windows cluster server. The cluster uses the same feature of the mainframe's multi-processor system. Another way windows systems can create redundancy is by using virtual machines. VMWare has a feature called Microsoft Cluster Service, which allows users to create a cluster of virtual machines on one physical windows system (or multiple physical machines). The virtual machines set up two different networks. They create a private network for communication in between the virtual machines and then a public network to control I/O services. The virtual machines also share storage to create concurrency so that if one fails, the other still has all of the data.
=== no downtime upgrades ===

A useful feature that mainframes have is the ability to hot-swap. Hot-swapping is the ability to swap out components of a computer/mainframe for new components with no downtime (i.e. the system continues to run through this process). Hot-swapping occurs when there is faulty hardware in one of the processors inside the mainframe. Technicians are able to swap out this component without the mainframe being turned off or crashing. Hot-swapping is also used when upgrading processors, memory and storage inside the mainframe. With the right software and setup (redundancy) a mainframe is able to be upgraded and/or repaired as is sees fit by adding and removing components such as hard drives and processors.

Using VMWare on a Windows system allows users to hot-add RAM and hot-plug adds a new virtual CPU to the virtualized system. Using these hot-adding and hot-plugging techniques the virtual computer can grow in size to be able to accept loads varying in size. Due to some circumstances with different CPU's and guest OS's, the virtual machine may have to restart and is unable to hot-add/hot-plug. For example, the virtual machine of a Windows Server 2008 ENT x64 OS allows you to hot-add memory but you must restart it to remove memory and to add/remove CPU.

In non-virtual systems, Windows coupled with the program Go-HotSwap can hot-plug CompactPCI components. CompactPCI components allow many different devices to be plugged into their slots (e.g. multiple SATA hard drives) which makes a Windows system with these technologies very modular. The Windows Server 2008 R2 Datacenter released in 2009 uses dynamic hardware partitioning. Dynamic hardware partitioning means that its hardware can be partitioned into separate hardware with processors and other components which allows for hot-swapping/hot-adding of these partitions where needed.

=== Backwards-Compatibility ===
Backwards-compatibility means that the newer software version can recognize what the old version write and how it work. It is a relationship between the two versions. If the new components provide all the functionality of the old one, we said that the new component is backwards compatible.In computer mainframe era, many applications are backwards compatible.For example,the code written 20 years ago in IBM System/360 can be run in latest mainframe (like zSeries, System/390 family,System z9,etc).This because that models in mainframe computer provide a combination of special hardware,special microcode and an emulation program to simulate the target system.(The IBM 7080 transistorized computer was backward compatible with all models of the IBM 705 vacuum tube computer.) Sometimes mainframe also need customers to halt the computer and download the emulation program.
In Windows OS,one method to implement backwards-compatibility is to add applications.Like Microsoft Windows Application Compatibility Toolkit.This application can make the platfrom to be compatible with most softwares from early version.The second method is the Windows Operating Systems usually have various subsystems.The software originally designed for older version or other OSs can be run in the subsystems.Such as Window NT, it has MS-DOS and Win16 subsystems.But Windows 7's backwards-compatibility is not very good.If kernel is different, the OSs can't be compatible with each other.But it doesn't mean that older programs won't run, virtualization will be used to make them run.The third method is to use shims to create the backwards-compatibility.Shims are just like the small libraries that can intercept the API, change parameters passed,handle and redirect the operations. In Windows OS,we can use shims to simulate the behaviors of old version OS for legacy softwares.

=== I/O and Resource Management ===
Throughput, unlike input and output, is the measurement of the number of calculations per second that a machine can preform. This is usually measured in FLOPS (floating point logical operations per second). It is impossible for one sole Windows machine to compete with a mainframe's throughput. Not only do mainframe processors have extremely high frequencies, but they also have a considerable amount of cores. This all changes, however, when computer clustering is introduced. In the recent years, IBM has constructed a clustered system called The Roadrunner that ranks third in the TOP500 supercomputer list as of June 2010.[http://hubpages.com/hub/Most-Powerful-Computers-In-The-World] It has a total of 60 connected units, over a thousand processors, and the capability of computing at a rate of 1.7 petaflops.

The question is, with such complex hardware, how is it possible for any sort of software to use this clustered system? Luckily, Windows has introduced an OS called Windows Compute Cluster Server, which provides the necessary software to allow the main computer to utilize the computing power of its cluster nodes. Windows mainly uses MS-MPI (Microsoft Message Passing Interface) to send messages via Ethernet to its other nodes.[http://webcache.googleusercontent.com/search?q=cache:EPlDExBxmDYJ:download.microsoft.com/download/9/e/d/9edcdeab-f1fb-4670-8914-c08c5c6f22a5/HPC_Overview.doc+Windows+Compute+Cluster+Server&cd=1&hl=en&ct=clnk&gl=ca&client=firefox-a] Developers can use this function because it automatically connects a given process to each node. Windows then can use its scheduler to determine which node receives each different job. It keeps track of each node, and shuts the job down once the output is received.

Today, clustering computers together with the intent of optimizing throughput is accomplished using grid computing. Grid computing shares the same basic idealisms of cluster computing, however, grids have the sole job of computing massive scale problems.[http://searchdatacenter.techtarget.com/definition/grid-computing] Each subsection of a problem is passed out to a compute node in the grid for it to be calculated. The one clear problem of this computational model is that the problem must have the ability to be broken down into several pieces for each compute node to work on. This style of high throughput computing can be used for problems such as high-energy physics, or biology models.

In general, however, the most popular solution to solve problems that require large throughput would be to construct a cluster model. Most businesses require the reliability of clusters, even though it sacrifices performance; there is no competition to the hight availability of a cluster server as compared to the grid model.[http://www.dba-oracle.com/real_application_clusters_rac_grid/grid_vs_clusters.htm]

== Conclusion ==
Again I don't think a conclusion is necessary unless its like one sentence. --[[User:Dkrutsko|Dkrutsko]] 23:43, 14 October 2010 (UTC)

== References ==

=== no downtime upgrades ===
Davis, David. "VMware vSphere hot-add RAM and hot-plug CPU." TechTarget. N.p., 09.15.2009. Web. 14 Oct 2010. <http://searchvmware.techtarget.com/tip/0,289483,sid179_gci1367631_mem1,00.html>.

"Windows Server 2008 R2 Datacenter." Windows Server 2008 R2. N.p., n.d. Web. 14 Oct 2010. <http://www.microsoft.com/windowsserver2008/en/us/2008-dc.aspx>.

"Go-HotSwap: CompactPCI Hot Swap." Jungo. Jungo Ltd, n.d. Web. 14 Oct 2010. <http://www.jungo.com/st/hotswap.html>.

=== Backwards-Compatibility
http://www.windows7news.com/2008/05/23/windows-7-to-break-backwards-compatibility/
http://computersight.com/computers/mainframe-computers/

COMP 3000 Essay 1 2010 Question 3

2010-10-15T05:02:25Z

Abown:

=Question=

To what extent do modern Windows systems provide mainframe-equivalent functionality? What about Windows coupled with add-on commercial products such as VMWare's virtualization and EMC's storage solutions? Explain.

=Answer=

Throughout the years, Windows has undergone some rather radical changes by modernizing existing technologies and providing innovation to existing features; this resulted in having functionality equivalent to that of a mainframe computer. However, although these changes have been extensive, Windows has not been particularly dominant when it comes to replacing modern mainframe systems.

== Mainframes ==

Mainframe systems have always had a good reputation for being used by large organizations in order to process thousands of small transactions. Whether these systems are used by the bank or by a police department, they possess several key features which make them exceedingly more powerful when compared to other systems. One of these features is an extensive and prolonged stability. This is a result of having tremendous redundancy and exception handling which prevents the entire system from shutting down, even if some components are inactive due to unforeseen circumstances. Because of this, mainframe computers are incredibly reliable when it comes to data storage and interoperability.

With this in mind, another neat feature that a mainframe possesses is the ability to hot swap components without taking the system offline. Consequently, components that are malfunctioning or require an upgrade can safely be replaced without endangering system stability. As a result mainframes gain a broad life spectrum as components can be upgraded individually without having to replace the entire system. Additionally, software written for these machines is extremely backwards compatible. The reason behind this is the fact that mainframe computers are fully virtualized. This is what allows a mainframe to run software that could have been written decades ago while still being able to run alongside modern software and hardware. In addition, this is part of the reason why mainframe computers are so secure, it is because they can use a combination of newer and older software as well as hardware to take years of innovation and combine it into one secure platform.

Unfortunately, all these features would mean nothing if the mainframe could not keep up with the data being sent and received. As a result, computers of this calibre must be able to have good I/O resource management as well as protect against bottlenecks. They do this by supporting powerful schedulers which ensure the fastest possible throughput for transaction processing [http://www.exforsys.com/tutorials/mainframe/mainframe-features.html]. Without this, you could continuously be upgrading components but suffer diminishing returns.

With so many features, how is Windows expected to keep up? The reality is Windows already supports most of these features. And when coupled with addon software such as VMWare and EMC storage solutions, the capabilities are even more astounding.

===Redundancy ===
A large feature of mainframes is their ability for redundancy. Mainframes produce redundancy by using the provider's off-site redundancy faeture. This feature lets the customer move all of their processes and applications onto the providers mainframe while the provider makes repairs on the customers system. Another way that mainframes create redundancy is their use of multi-processors that share the same memory. If one processor dies, the rest of the processors still keep all of the cache. There are multiple ways windows systems can create this redundancy feature that mainframes have. The first way windows systems can create this is by creating a windows cluster server. The cluster uses the same feature of the mainframe's multi-processor system. Another way windows systems can create redundancy is by using virtual machines. VMWare has a feature called Microsoft Cluster Service, which allows users to create a cluster of virtual machines on one physical windows system (or multiple physical machines). The virtual machines set up two different networks. They create a private network for communication in between the virtual machines and then a public network to control I/O services. The virtual machines also share storage to create concurrency so that if one fails, the other still has all of the data.
=== no downtime upgrades ===

A useful feature that mainframes have is the ability to hot-swap. Hot-swapping is the ability to swap out components of a computer/mainframe for new components with no downtime (i.e. the system continues to run through this process). Hot-swapping occurs when there is faulty hardware in one of the processors inside the mainframe. Technicians are able to swap out this component without the mainframe being turned off or crashing. Hot-swapping is also used when upgrading processors, memory and storage inside the mainframe. With the right software and setup (redundancy) a mainframe is able to be upgraded and/or repaired as is sees fit by adding and removing components such as hard drives and processors.

Using VMWare on a Windows system allows users to hot-add RAM and hot-plug adds a new virtual CPU to the virtualized system. Using these hot-adding and hot-plugging techniques the virtual computer can grow in size to be able to accept loads varying in size. Due to some circumstances with different CPU's and guest OS's, the virtual machine may have to restart and is unable to hot-add/hot-plug. For example, the virtual machine of a Windows Server 2008 ENT x64 OS allows you to hot-add memory but you must restart it to remove memory and to add/remove CPU.

In non-virtual systems, Windows coupled with the program Go-HotSwap can hot-plug CompactPCI components. CompactPCI components allow many different devices to be plugged into their slots (e.g. multiple SATA hard drives) which makes a Windows system with these technologies very modular. The Windows Server 2008 R2 Datacenter released in 2009 uses dynamic hardware partitioning. Dynamic hardware partitioning means that its hardware can be partitioned into separate hardware with processors and other components which allows for hot-swapping/hot-adding of these partitions where needed.

=== Backwards-Compatibility ===
Backwards-compatibility means that the newer software version can recognize what the old version write and how it work. It is a relationship between the two versions. If the new components provide all the functionality of the old one, we said that the new component is backwards compatible.In computer mainframe era, many applications are backwards compatible.For example,the code written 20 years ago in IBM System/360 can be run in latest mainframe (like zSeries, System/390 family,System z9,etc).This because that models in mainframe computer provide a combination of special hardware,special microcode and an emulation program to simulate the target system.(The IBM 7080 transistorized computer was backward compatible with all models of the IBM 705 vacuum tube computer.) Sometimes mainframe also need customers to halt the computer and download the emulation program.
In Windows OS,one method to implement backwards-compatibility is to add applications.Like Microsoft Windows Application Compatibility Toolkit.This application can make the platfrom to be compatible with most softwares from early version.The second method is the Windows Operating Systems usually have various subsystems.The software originally designed for older version or other OSs can be run in the subsystems.Such as Window NT, it has MS-DOS and Win16 subsystems.But Windows 7's backwards-compatibility is not very good.If kernel is different, the OSs can't be compatible with each other.But it doesn't mean that older programs won't run, virtualization will be used to make them run.The third method is to use shims to create the backwards-compatibility.Shims are just like the small libraries that can intercept the API, change parameters passed,handle and redirect the operations. In Windows OS,we can use shims to simulate the behaviors of old version OS for legacy softwares.
=== Thin Client Terminals ===

=== 64 Bit Support ===

=== Better Multi-Core support ===

=== Mass Storage Hot Swapping ===

== Addon Software ==

=== Virtualization ===

- VMWare 
- Virtual Box

=== Backup Solutions ===

- EMC Storage solutions 
- Carbonite

== Conclusion ==
Again I don't think a conclusion is necessary unless its like one sentence. --[[User:Dkrutsko|Dkrutsko]] 23:43, 14 October 2010 (UTC)

== References ==

=== no downtime upgrades ===
Davis, David. "VMware vSphere hot-add RAM and hot-plug CPU." TechTarget. N.p., 09.15.2009. Web. 14 Oct 2010. <http://searchvmware.techtarget.com/tip/0,289483,sid179_gci1367631_mem1,00.html>.

"Windows Server 2008 R2 Datacenter." Windows Server 2008 R2. N.p., n.d. Web. 14 Oct 2010. <http://www.microsoft.com/windowsserver2008/en/us/2008-dc.aspx>.

"Go-HotSwap: CompactPCI Hot Swap." Jungo. Jungo Ltd, n.d. Web. 14 Oct 2010. <http://www.jungo.com/st/hotswap.html>.

=== Backwards-Compatibility
http://www.windows7news.com/2008/05/23/windows-7-to-break-backwards-compatibility/
http://computersight.com/computers/mainframe-computers/

Talk:COMP 3000 Essay 1 2010 Question 3

2010-10-14T15:14:02Z

Abown: /* High input/output */

Talk:COMP 3000 Essay 1 2010 Question 3

2010-10-14T15:12:29Z

Abown: /* High input/output */

COMP 3000 Essay 1 2010 Question 3

2010-10-13T18:08:22Z

Abown: /* High input/output */

=Question=

To what extent do modern Windows systems provide mainframe-equivalent functionality? What about Windows coupled with add-on commercial products such as VMWare's virtualization and EMC's storage solutions? Explain.

=Answer=

added introduction points and sections for each paragraph so you guys can edit one paragraph at a time instead of the whole document. If you want to claim a certain paragram just put your name into the section first. ~ Andrew (abown2@connect.carleton.ca) 12:00 10th of October 2010
== Introduction ==

Main Aspects of mainframes:
* redundancy which enables high reliability and security
* high input/output
* backwards-compatibility with legacy software
* support massive throughput
* Systems run constantly so they can be hot upgraded
http://www.exforsys.com/tutorials/mainframe/mainframe-features.html

Linking sentence about how windows can duplicate mainframe functionality.

here's the introduction ~ Abown (11:12 pm, October 12th 2010)

Mainframes have been always used for large corporations to process thousands of small transactions, but what strengths allow for mainframes to be useful in their purpose. Mainframes are extremely useful in business because they are designed to run without downtime. This is achieved by have tremendous redundancy which allows for mainframes to be extremely reliable which gives security when concerning data loss due to downtime. Also mainframes can be upgraded without taking the system down which allows for repairs, which further increase reliability. But after upgrading a mainframe the software does not change so they offer the features of backwards compatibility through virtualization so software never needs to be replaced, it just it processed quicker. But computers are only able to run as fast as the data they are receiving so mainframes support high input/output so that the mainframe is always being utilized. To make sure mainframes are utilized to their fullest they support powerful schedulers which ensure the fastest throughput for processing transactions as possible.[http://www.exforsys.com/tutorials/mainframe/mainframe-features.html] With so many features how is Windows based system supposed to compete with a mainframe? The fact of the matter is their are features in Windows and software solutions which can duplicate these features in a Windows environment. Be it redundancy, real-time upgrading, virtualization, high input/output or utilizing resources.

== History ==
Before comparing Windows systems and mainframes, the history of what mainframes were used for and where they came from must be understood. The first official mainframe computer was the UNIVAC I. [http://www.vikingwaters.com/htmlpages/MFHistory.htm] It was designed for the U.S. Census Bureau by J. Presper Eckert and John Mauchly. [http://www.thocp.net/hardware/univac.htm]. By this point in history, there were no personal computers, and the only people who could afford a computer were massive businesses. The main functionality of these mainframes were to calculate company payrolls, sales records, analyze sales performance, and store all company information. 
[[User:Achamney|Achamney]] 01:30, 12 October 2010 (UTC)

This doesn't seem to actually be pertinent to the question at hand. Question does not have any indication of the need to provide a history. [[User:Abown|Andrew Bown]] 11:16, 12 October 2010

== Redundancy ==
[[User:Nshires|Nshires]] 04:10, 13 October 2010 (UTC)
A large feature of mainframes is their ability for redundancy. Mainframes produce redundancy by using the provider's off-site redundancy faeture. This feature lets the customer move all of their processes and applications onto the providers mainframe while the provider makes repairs on the customers system. Another way that mainframes create redundancy is their use of multi-processors that share the same memory. If one processor dies, the rest of the processors still keep all of the cache. There are multiple ways windows systems can create this redundancy feature that mainframes have. The first way windows systems can create this is by creating a windows cluster server. The cluster uses the same feature of the mainframe's multi-processor system. Another way windows systems can create redundancy is by using virtual machines. VMWare has a feature called Microsoft Cluster Service, which allows users to create a cluster of virtual machines on one physical windows system (or multiple physical machines). The virtual machines set up two different networks. They create a private network for communication in between the virtual machines and then a public network to control I/O services. The virtual machines also share storage to create concurrency so that if one fails, the other still has all of the data.

(this is what I've gotten out of some researching so far, comments and any edits/suggestions if I'm on the right track or not are greatly apreciated :) )
*note: This is the second time I have written this, make sure to save whatever you edit in notepad or whatever first so that you don't lose everything*

link to VMWare's cluster virtualization http://www.vmware.com/pdf/vsphere4/r40/vsp_40_mscs.pdf

[[User:Nshires|Nshires]] 04:10, 13 October 2010 (UTC)

== hot swapping ==
[[User:Nshires|Nshires]] 16:47, 13 October 2010 (UTC)
Another useful feature that mainframes have is the ability to hot-swap. Hot-swapping occurs when there is faulty hardware in one of the processors inside the mainframe and technicians are able to swap out this component without the mainframe being turned off or crashing. Hot-swapping is also used when upgrading processors inside the mainframe. With the right software and setup (redundancy) a mainframe is able to upgrade and/or repair their mainframe as they see fit. Using VMWare on a Windows system allows users to hot-add RAM and hot-plug adds a new virtual CPU to the virtualized system. Using these hot-adding and hot-plugging techniques the virtual computer can grow in size to be able to accept loads varying in size. In non-virtual systems, Windows coupled with the program Go-HotSwap can hot-plug CompactPCI components. CompactPCI components allow many different devices to be plugged into their slots (e.g. multiple SATA hard drives) which makes a Windows system with these technologies very modular.

These are the concepts I've been able to figure out so far about hot-swapping/hot-upgrading, feel free to add/edit and what-not!

Sources:
http://searchvmware.techtarget.com/tip/0,289483,sid179_gci1367631,00.html
http://www.jungo.com/st/hotswap_windows.html
[[User:Nshires|Nshires]] 16:47, 13 October 2010 (UTC)

== backwards-compatibility ==
Backwards-compatibility means that the newer software version can recognize what the old version write and how it work. It is a relationship between the two versions. If the new components provide all the functionality of the old one, we said that the new component is backwards compatible.In computer mainframe era, many applications are backwards compatible.For example,the code written 20 years ago in IBM System/360 can be run in latest mainframe (like zSeries, System/390 family,System z9,etc).This because that models in mainframe computer provide a combination of special hardware,special microcode and an emulation program to simulate the target system.(The IBM 7080 transistorized computer was backward compatible with all models of the IBM 705 vacuum tube computer.) Sometimes mainframe also need customers to halt the computer and download the emulation program.

In Windows OS,one method to implement backward-compatibility is to add applications.Then the platfrom can be compatible with most softwares from early version.The other method is the Windows Operating Systems usually have various subsystems.The software originally designed for older version or other OSs can be run in the subsystems.Such as Window NT, it has MS-DOS and Win16 subsystems.But Windows 7's backwards-compatibility is not very good.If kernel is different, the OSs can't be compatible with each other.But it doesn't mean that older programs won't run, virtualization will be used to make them run.

--[[User:Zhangqi|Zhangqi]] 08:34, 13 October 2010 (UTC)

ps. I didn't find perfect resources,just these.If you guys think any opinion is not correct,plz edit it or give suggestions :)

http://www.windows7news.com/2008/05/23/windows-7-to-break-backwards-compatibility/

http://computersight.com/computers/mainframe-computers/

Hey, this sounds really good, I'd add an example where you say 'one method to implement backward-compatibility is to add applications'.
And I did a little research and I found another way to create backwards compatibility using shims: http://en.wikipedia.org/wiki/Shim_%28computing%29
it pretty much intercepts the calls and changes them so that the old program can run on a new system.
Good Work, [[User:Nshires|Nshires]] 16:56, 13 October 2010 (UTC)

== High input/output ==
~Andrew Bown (October 13 2:08) I'll write this paragraph.

== massive throughput ==

== Conclusion ==

== References ==

COMP 3000 Essay 1 2010 Question 3

2010-10-13T03:20:44Z

Abown:

=Question=

To what extent do modern Windows systems provide mainframe-equivalent functionality? What about Windows coupled with add-on commercial products such as VMWare's virtualization and EMC's storage solutions? Explain.

=Answer=

added introduction points and sections for each paragraph so you guys can edit one paragraph at a time instead of the whole document. If you want to claim a certain paragram just put your name into the section first. ~ Andrew (abown2@connect.carleton.ca) 12:00 10th of October 2010
== Introduction ==

Main Aspects of mainframes:
* redundancy which enables high reliability and security
* high input/output
* backwards-compatibility with legacy software
* support massive throughput
* Systems run constantly so they can be hot upgraded
http://www.exforsys.com/tutorials/mainframe/mainframe-features.html

Linking sentence about how windows can duplicate mainframe functionality.

here's the introduction ~ Abown (11:12 pm, October 12th 2010)

Mainframes have been always used for large corporations to process thousands of small transactions, but what strengths allow for mainframes to be useful in their purpose. Mainframes are extremely useful in business because they are designed to run without downtime. This is achieved by have tremendous redundancy which allows for mainframes to be extremely reliable which gives security when concerning data loss due to downtime. Also mainframes can be upgraded without taking the system down which allows for repairs, which further increase reliability. But after upgrading a mainframe the software does not change so they offer the features of backwards compatibility through virtualization so software never needs to be replaced, it just it processed quicker. But computers are only able to run as fast as the data they are receiving so mainframes support high input/output so that the mainframe is always being utilized. To make sure mainframes are utilized to their fullest they support powerful schedulers which ensure the fastest throughput for processing transactions as possible.[http://www.exforsys.com/tutorials/mainframe/mainframe-features.html] With so many features how is Windows based system supposed to compete with a mainframe? The fact of the matter is their are features in Windows and software solutions which can duplicate these features in a Windows environment. Be it redundancy, real-time upgrading, virtualization, high input/output or utilizing resources.

== History ==
Before comparing Windows systems and mainframes, the history of what mainframes were used for and where they came from must be understood. The first official mainframe computer was the UNIVAC I. [http://www.vikingwaters.com/htmlpages/MFHistory.htm] It was designed for the U.S. Census Bureau by J. Presper Eckert and John Mauchly. [http://www.thocp.net/hardware/univac.htm]. By this point in history, there were no personal computers, and the only people who could afford a computer were massive businesses. The main functionality of these mainframes were to calculate company payrolls, sales records, analyze sales performance, and store all company information. 
[[User:Achamney|Achamney]] 01:30, 12 October 2010 (UTC)

This doesn't seem to actually be pertinent to the question at hand. Question does not have any indication of the need to provide a history. [[User:Abown|Andrew Bown]] 11:16, 12 October 2010

== Redundancy ==

== hot upgrades ==

== backwards-compatibility ==

== High input/output ==

== massive throughput ==

== Conclusion ==

== References ==

COMP 3000 Essay 1 2010 Question 3

2010-10-13T03:17:04Z

Abown: /* History */

=Question=

To what extent do modern Windows systems provide mainframe-equivalent functionality? What about Windows coupled with add-on commercial products such as VMWare's virtualization and EMC's storage solutions? Explain.

=Answer=

added introduction points and sections for each paragraph so you guys can edit one paragraph at a time instead of the whole document. If you want to claim a certain paragram just put your name into the section first. ~ Andrew (abown2@connect.carleton.ca) 12:00 10th of October 2010
== Introduction ==

Main Aspects of mainframes:
* redundancy which enables high reliability and security
* high input/output
* backwards-compatibility with legacy software
* support massive throughput
* Systems run constantly so they can be hot upgraded
http://www.exforsys.com/tutorials/mainframe/mainframe-features.html

Linking sentence about how windows can duplicate mainframe functionality.

here's the introduction ~ Abown (11:12 pm, October 12th 2010)

Mainframes have been always used for large corporations to process thousands of small transactions, but what strengths allow for mainframes to be useful in their purpose. Mainframes are extremely useful in business because they are designed to run without downtime. This is achieved by have tremendous redundancy which allows for mainframes to be extremely reliable which gives security when concerning data loss due to downtime. Also mainframes can be upgraded without taking the system down which allows for repairs, which further increase reliability. But after upgrading a mainframe the software does not change so they offer the features of backwards compatibility through virtualization so software never needs to be replaced, it just it processed quicker. But computers are only able to run as fast as the data they are receiving so mainframes support high input/output so that the mainframe is always being utilized. To make sure mainframes are utilized to their fullest they support powerful schedulers which ensure the fastest throughput for processing transactions as possible.[http://www.exforsys.com/tutorials/mainframe/mainframe-features.html] With so many features how is Windows based system supposed to compete with a mainframe? The fact of the matter is their are features in Windows and software solutions which can duplicate these features in a Windows environment. Be it redundancy, real-time upgrading, virtualization, high input/output or utilizing resources.

== History ==
Before comparing Windows systems and mainframes, the history of what mainframes were used for and where they came from must be understood. The first official mainframe computer was the UNIVAC I. [http://www.vikingwaters.com/htmlpages/MFHistory.htm] It was designed for the U.S. Census Bureau by J. Presper Eckert and John Mauchly. [http://www.thocp.net/hardware/univac.htm]. By this point in history, there were no personal computers, and the only people who could afford a computer were massive businesses. The main functionality of these mainframes were to calculate company payrolls, sales records, analyze sales performance, and store all company information. 
[[User:Achamney|Achamney]] 01:30, 12 October 2010 (UTC)

This doesn't seem to actually be pertinent to the question at hand. Question does not have any indication of the need to provide a history. [[User:Abown|Andrew Bown]] 11:16, 12 October 2010

== Redundancy ==

== High input/output ==

== backwards-compatibility ==

== massive throughput ==

== hot upgrades ==

== Conclusion ==

== References ==

COMP 3000 Essay 1 2010 Question 3

2010-10-13T03:16:44Z

Abown: /* History */

=Question=

To what extent do modern Windows systems provide mainframe-equivalent functionality? What about Windows coupled with add-on commercial products such as VMWare's virtualization and EMC's storage solutions? Explain.

=Answer=

added introduction points and sections for each paragraph so you guys can edit one paragraph at a time instead of the whole document. If you want to claim a certain paragram just put your name into the section first. ~ Andrew (abown2@connect.carleton.ca) 12:00 10th of October 2010
== Introduction ==

Main Aspects of mainframes:
* redundancy which enables high reliability and security
* high input/output
* backwards-compatibility with legacy software
* support massive throughput
* Systems run constantly so they can be hot upgraded
http://www.exforsys.com/tutorials/mainframe/mainframe-features.html

Linking sentence about how windows can duplicate mainframe functionality.

here's the introduction ~ Abown (11:12 pm, October 12th 2010)

Mainframes have been always used for large corporations to process thousands of small transactions, but what strengths allow for mainframes to be useful in their purpose. Mainframes are extremely useful in business because they are designed to run without downtime. This is achieved by have tremendous redundancy which allows for mainframes to be extremely reliable which gives security when concerning data loss due to downtime. Also mainframes can be upgraded without taking the system down which allows for repairs, which further increase reliability. But after upgrading a mainframe the software does not change so they offer the features of backwards compatibility through virtualization so software never needs to be replaced, it just it processed quicker. But computers are only able to run as fast as the data they are receiving so mainframes support high input/output so that the mainframe is always being utilized. To make sure mainframes are utilized to their fullest they support powerful schedulers which ensure the fastest throughput for processing transactions as possible.[http://www.exforsys.com/tutorials/mainframe/mainframe-features.html] With so many features how is Windows based system supposed to compete with a mainframe? The fact of the matter is their are features in Windows and software solutions which can duplicate these features in a Windows environment. Be it redundancy, real-time upgrading, virtualization, high input/output or utilizing resources.

== History ==
Before comparing Windows systems and mainframes, the history of what mainframes were used for and where they came from must be understood. The first official mainframe computer was the UNIVAC I. [http://www.vikingwaters.com/htmlpages/MFHistory.htm] It was designed for the U.S. Census Bureau by J. Presper Eckert and John Mauchly. [http://www.thocp.net/hardware/univac.htm]. By this point in history, there were no personal computers, and the only people who could afford a computer were massive businesses. The main functionality of these mainframes were to calculate company payrolls, sales records, analyze sales performance, and store all company information. 
[[User:Achamney|Achamney]] 01:30, 12 October 2010 (UTC)
This doesn't seem to actually be pertinent to the question at hand. Question does not have any indication of the need to provide a history. [[User:Abown|Andrew Bown]] 11:16, 12 October 2010

== Redundancy ==

== High input/output ==

== backwards-compatibility ==

== massive throughput ==

== hot upgrades ==

== Conclusion ==

== References ==

COMP 3000 Essay 1 2010 Question 3

2010-10-13T03:12:53Z

Abown: /* Introduction */

=Question=

To what extent do modern Windows systems provide mainframe-equivalent functionality? What about Windows coupled with add-on commercial products such as VMWare's virtualization and EMC's storage solutions? Explain.

=Answer=

added introduction points and sections for each paragraph so you guys can edit one paragraph at a time instead of the whole document. If you want to claim a certain paragram just put your name into the section first. ~ Andrew (abown2@connect.carleton.ca) 12:00 10th of October 2010
== Introduction ==

Main Aspects of mainframes:
* redundancy which enables high reliability and security
* high input/output
* backwards-compatibility with legacy software
* support massive throughput
* Systems run constantly so they can be hot upgraded
http://www.exforsys.com/tutorials/mainframe/mainframe-features.html

Linking sentence about how windows can duplicate mainframe functionality.

here's the introduction ~ Abown (11:12 pm, October 12th 2010)

Mainframes have been always used for large corporations to process thousands of small transactions, but what strengths allow for mainframes to be useful in their purpose. Mainframes are extremely useful in business because they are designed to run without downtime. This is achieved by have tremendous redundancy which allows for mainframes to be extremely reliable which gives security when concerning data loss due to downtime. Also mainframes can be upgraded without taking the system down which allows for repairs, which further increase reliability. But after upgrading a mainframe the software does not change so they offer the features of backwards compatibility through virtualization so software never needs to be replaced, it just it processed quicker. But computers are only able to run as fast as the data they are receiving so mainframes support high input/output so that the mainframe is always being utilized. To make sure mainframes are utilized to their fullest they support powerful schedulers which ensure the fastest throughput for processing transactions as possible.[http://www.exforsys.com/tutorials/mainframe/mainframe-features.html] With so many features how is Windows based system supposed to compete with a mainframe? The fact of the matter is their are features in Windows and software solutions which can duplicate these features in a Windows environment. Be it redundancy, real-time upgrading, virtualization, high input/output or utilizing resources.

== History ==
Before comparing Windows systems and mainframes, the history of what mainframes were used for and where they came from must be understood. The first official mainframe computer was the UNIVAC I. [http://www.vikingwaters.com/htmlpages/MFHistory.htm] It was designed for the U.S. Census Bureau by J. Presper Eckert and John Mauchly. [http://www.thocp.net/hardware/univac.htm]. By this point in history, there were no personal computers, and the only people who could afford a computer were massive businesses. The main functionality of these mainframes were to calculate company payrolls, sales records, analyze sales performance, and store all company information. 
[[User:Achamney|Achamney]] 01:30, 12 October 2010 (UTC)

== Redundancy ==

== High input/output ==

== backwards-compatibility ==

== massive throughput ==

== hot upgrades ==

== Conclusion ==

== References ==

COMP 3000 Essay 1 2010 Question 3

2010-10-13T03:11:25Z

Abown: /* Introduction */ complete

=Question=

To what extent do modern Windows systems provide mainframe-equivalent functionality? What about Windows coupled with add-on commercial products such as VMWare's virtualization and EMC's storage solutions? Explain.

=Answer=

added introduction points and sections for each paragraph so you guys can edit one paragraph at a time instead of the whole document. If you want to claim a certain paragram just put your name into the section first. ~ Andrew (abown2@connect.carleton.ca) 12:00 10th of October 2010
== Introduction ==

Main Aspects of mainframes:
* redundancy which enables high reliability and security
* high input/output
* backwards-compatibility with legacy software
* support massive throughput
* Systems run constantly so they can be hot upgraded
http://www.exforsys.com/tutorials/mainframe/mainframe-features.html

Linking sentence about how windows can duplicate mainframe functionality.

here's the introduction although

Mainframes have been always used for large corporations to process thousands of small transactions, but what strengths allow for mainframes to be useful in their purpose. Mainframes are extremely useful in business because they are designed to run without downtime. This is achieved by have tremendous redundancy which allows for mainframes to be extremely reliable which gives security when concerning data loss due to downtime. Also mainframes can be upgraded without taking the system down which allows for repairs, which further increase reliability. But after upgrading a mainframe the software does not change so they offer the features of backwards compatibility through virtualization so software never needs to be replaced, it just it processed quicker. But computers are only able to run as fast as the data they are receiving so mainframes support high input/output so that the mainframe is always being utilized. To make sure mainframes are utilized to their fullest they support powerful schedulers which ensure the fastest throughput for processing transactions as possible.[http://www.exforsys.com/tutorials/mainframe/mainframe-features.html] With so many features how is Windows based system supposed to compete with a mainframe? The fact of the matter is their are features in Windows and software solutions which can duplicate these features in a Windows environment. Be it redundancy, real-time upgrading, virtualization, high input/output or utilizing resources.

== History ==
Before comparing Windows systems and mainframes, the history of what mainframes were used for and where they came from must be understood. The first official mainframe computer was the UNIVAC I. [http://www.vikingwaters.com/htmlpages/MFHistory.htm] It was designed for the U.S. Census Bureau by J. Presper Eckert and John Mauchly. [http://www.thocp.net/hardware/univac.htm]. By this point in history, there were no personal computers, and the only people who could afford a computer were massive businesses. The main functionality of these mainframes were to calculate company payrolls, sales records, analyze sales performance, and store all company information. 
[[User:Achamney|Achamney]] 01:30, 12 October 2010 (UTC)

== Redundancy ==

== High input/output ==

== backwards-compatibility ==

== massive throughput ==

== hot upgrades ==

== Conclusion ==

== References ==

Talk:COMP 3000 Essay 1 2010 Question 3

2010-10-11T15:29:24Z

Abown:

Talk:COMP 3000 Essay 1 2010 Question 3

2010-10-10T18:35:27Z

Abown: added some extra information

COMP 3000 Essay 1 2010 Question 3

2010-10-10T17:41:59Z

Abown: /* Introduction */

=Question=

To what extent do modern Windows systems provide mainframe-equivalent functionality? What about Windows coupled with add-on commercial products such as VMWare's virtualization and EMC's storage solutions? Explain.

=Answer=

added introduction points and sections for each paragraph so you guys can edit one paragraph at a time instead of the whole document. If you want to claim a certain paragram just put your name into the section first. ~ Andrew (abown2@connect.carleton.ca) 12:00 10th of October 2010
== Introduction ==

Main Aspects of mainframes:
* redundancy which enables high reliability and security
* high input/output
* backwards-compatibility with legacy software
* support massive throughput
* Systems run constantly so they can be hot upgraded
http://publib.boulder.ibm.com/infocenter/zos/basics/topic/com.ibm.zos.zmainframe/zconc_mfhardware.htm

Linking sentence about how windows can duplicate mainframe functionality.

== Redundancy ==

== High input/output ==

== backwards-compatibility ==

== massive throughput ==

== hot upgrades ==

== Conclusion ==

== References ==

COMP 3000 Essay 1 2010 Question 3

2010-10-10T17:41:45Z

Abown: /* Introduction */

=Question=

To what extent do modern Windows systems provide mainframe-equivalent functionality? What about Windows coupled with add-on commercial products such as VMWare's virtualization and EMC's storage solutions? Explain.

=Answer=

added introduction points and sections for each paragraph so you guys can edit one paragraph at a time instead of the whole document. If you want to claim a certain paragram just put your name into the section first. ~ Andrew (abown2@connect.carleton.ca) 12:00 10th of October 2010
== Introduction ==

Main Aspects of mainframes:
* redundancy which enables high reliability and security
* high input/output
* backwards-compatibility with legacy software
* support massive throughput
* Systems run constantly so they can be hot upgraded
http://publib.boulder.ibm.com/infocenter/zos/basics/topic/com.ibm.zos.zmainframe/zconc_mfhardware.htm
Linking sentence about how windows can duplicate mainframe functionality.

== Redundancy ==

== High input/output ==

== backwards-compatibility ==

== massive throughput ==

== hot upgrades ==

== Conclusion ==

== References ==

COMP 3000 Essay 1 2010 Question 3

2010-10-10T16:05:30Z

Abown: added sections to answer

=Question=

To what extent do modern Windows systems provide mainframe-equivalent functionality? What about Windows coupled with add-on commercial products such as VMWare's virtualization and EMC's storage solutions? Explain.

=Answer=

added introduction points and sections for each paragraph so you guys can edit one paragraph at a time instead of the whole document. If you want to claim a certain paragram just put your name into the section first. ~ Andrew (abown2@connect.carleton.ca) 12:00 10th of October 2010
== Introduction ==

Main Aspects of mainframes:
* redundancy which enables high reliability and security
* high input/output
* backwards-compatibility with legacy software
* support massive throughput
* Systems run constantly so they can be hot upgraded
(unfortunately these aspects seem to be such common knowledge that I can't get a good reference for it)
Linking sentence about how windows can duplicate mainframe functionality.

== Redundancy ==

== High input/output ==

== backwards-compatibility ==

== massive throughput ==

== hot upgrades ==

== Conclusion ==

== References ==

Talk:COMP 3000 Essay 1 2010 Question 3

2010-10-10T15:03:29Z

Abown: moved link from answer to discussion

Talk:COMP 3000 Essay 1 2010 Question 3

2010-10-10T15:00:57Z

Abown:

Talk:COMP 3000 Essay 1 2010 Question 3

2010-10-07T17:57:10Z

Abown: /* Group 3 */ added paper

Talk:COMP 3000 Essay 1 2010 Question 3

2010-10-06T02:48:17Z

Abown: added email address to discussion

== Group 3 ==
Here's my email I'll add some of the stuff I find soon I'm just saving the question for last.
Andrew Bown(abown2@connect.carleton.ca)