COMP 3000 Essay 2 2010 Question 1

An Analysis of Linux Scalability to Many Cores

Authors: Silas Boyd-Wickizer, Austin T. Clements, Yandong Mao, Aleksey Pesterev, M. Frans Kaashoek, Robert Morris and Nickolai Zeldovich.

Affiliates: MIT CSAIL

Background Concepts

memcached: Section 3.2

memcached is an in-memory hash table server. One instance of memcached running on many different cores is bottlenecked by an internal lock, which is avoided by the MIT team by running one instance per core. Clients each connect to a single instance of memcached, allowing the server to simulate parallelism without needing to make major changes to the application or kernel. With few requests, memcached spends 80% of its time in the kernel on one core, mostly processing packets.¹

Apache: Section 3.3

Apache is a web server that has been used in previous Linux scalability studies. In the case of this study, Apache has been configured to run a separate process on each core. Each process, in turn, has multiple threads (making it a perfect example of parallel programming). Each process uses one of their threads to accepting incoming connections and others are used to process these connections. On a single core processor, Apache spends 60% of its execution time in the kernel.¹

gmake: Section 3.5

gmake is an unofficial default benchmark in the Linux community which is used in this paper to build the Linux kernel. gmake takes a file called a makefile and processes its recipes for the requisite files to determine how and when to remake or recompile code. With a simple command -j or --jobs, gmake can process many of these recipes in parallel. Since gmake creates more processes than cores, it can make proper use of multiple cores to process the recipes.⁵ Since gmake involves much reading and writing, in order to prevent bottlenecks due to the filesystem or hardware, the test cases use an in-memory filesystem tmpfs, which gives them a backdoor around the bottlenecks for testing purposes. In addition to this, gmake is limited in scalability due to the serial processes that run at the beginning and end of its execution, which limits its scalability to a small degree. gmake spends much of its execution time with its compiler, processing the recipes and recompiling code, but still spend 7.6% of its time in system time.¹

Research problem

As technology progresses, the number of core a main processor can have is increasing at an impressive rate. Soon personal computers will have so many cores that scalability will be an issue. There has to be a way that standard user level Linux kernel will scale with a 48-core system¹. The problem with a standard Linux OS is that they are not designed for massive scalability, which will soon prove to be a problem. The issue with scalability is that a solo core will perform much more work compared to a single core working with 47 other cores. Although traditional logic states that the situation makes sense because there are 48 cores dividing the work, the information should be processed as fast as possible with each core doing as much work as possible.

To fix those scalability issues, it is necessary to focus on three major areas: the Linux kernel, user level design and how applications use kernel services. The Linux kernel can be improved by optimizing sharing and use the current advantages of recent improvement to scalability features. On the user level design, applications can be improved so that there is more focus on parallelism since some programs have not implemented those improved features. The final aspect of improving scalability is how an application uses kernel services to better share resources so that different aspects of the program are not conflicting over the same services. All of the bottlenecks are found easily and actually only take simple changes to correct or avoid.¹

This research uses a foundation of previous research discovered during the development of scalability in UNIX systems. The major developments from shared memory machines² and wait-free synchronization to fast message passing ended up creating a base set of techniques, which can be used to improve scalability. These techniques have been incorporated in all major operating system including Linux, Mac OS X and Windows. Linux has been improved with kernel subsystems, such as Read-Copy-Update, which is an algorithm that is used to avoid locks and atomic instructions which affect scalability.³ There is an excellent base of research on Linux scalability studies that have already been written, on which this research paper can model its testing standards. These papers include research on improving scalability on a 32-core machine.⁴ In addition, the base of studies can be used to improve the results of these experiments by learning from the previous results. This research may also aid in identifying bottlenecks which speed up creating solutions for those problems.

Contribution

MOSBENCH & 16 scalability improvements

MOSBENCH is a set of application available through MIT. They are designed to measure scalability: "It consists of applications that previous work has shown not to scale well on Linux and applications that are designed for parallel execution and are kernel intensive."⁶ Overall, there were 16 scalability problems encountered by MOSBENCH applications within the scope of the paper and each problem was fixed. The fixes add 2617 lines of code and remove 385 lines of code from Linux.¹

Techniques to improve scalability

What hinders scalability: Section 4.1

Amdahl's Law states that a parallel program can only be sped up by the inverse of the portion of the program that cannot be made parallel, so, for example, if a program is 50% non-parallel, then at most the program can be sped up to twice the speed using parallelism. So the more serial a program is, the less capability it has for scalability. The main problems found within the MOSBENCH applications that cause serialized interactions were locking of shared data structures, writing to shared memory, competing for space in shared hardware caches, competing for shared hardware resources, and the lack of tasks leading to idle cores. These problems become more evident as more cores are added to the system. The team behind the paper came up with some solutions that either fixed these problems or avoided most, if not all, of the bottlenecks that occurred.¹

Multicore packet processing: Section 4.2

Linux packet processing technique requires the packets to travel along several queues before it finally becomes available for the application to use. This technique works well for most general socket applications. In recent kernel releases Linux takes advantage of multiple hardware queues (when available on the given network interface) or Receive Packet Steering⁸ to direct packet flow onto different cores for processing. Or even go as far as directing packet flow to the core on which the application is running using Receive Flow Steering⁹ for even better performance. Linux also attempts to increase performance using a sampling technique where it checks every 20th outgoing packet and directs flow based on its hash. This poses a problem for short lived connections like those associated with Apache since there is great potential for packets to be misdirected.

In general, this technique performs poorly when there are numerous open connections spread across multiple cores due to mutex (mutual exclusion) delays and cache misses. In such scenarios, it's better to process all connections, with associated packets and queues, on one core to avoid said issues. The patched kernel's implementation proposed in this article uses multiple hardware queues (which can be accomplished through Receive Packet Sharing) to direct all packets from a given connection to the same core. In turn, Apache is modified to only accept connections if the thread dedicated to processing them is on the same core. If the current core's queue is found to be empty, it will attempt to obtain work from queues located on different cores.¹ This configuration is ideal for numerous short connections as all the work for them in accomplished quickly on one core avoiding unnecessary mutex delays associated with packet queues and inter-core cache misses.

Sloppy counters: Section 4.3

Bottlenecks were encountered when the applications undergoing testing were referencing and updating shared counters for multiple cores. The solution in the paper is to use sloppy counters, which gets each core to track its own separate counts of references and uses a central shared counter to keep all counts on track. This is ideal because each core updates its counts by modifying its per-core counter, usually only needing access to its own local cache, cutting down on waiting for locks or serialization. Sloppy counters are also backwards-compatible with existing shared-counter code, making its implementation much easier to accomplish. The main disadvantages of the sloppy counters are that in situations where object de-allocation occurs often, because the de-allocation itself is an expensive operation, and the counters use up space proportional to the number of cores.¹

Lock-free comparison & Avoiding unnecessary locking: Section 4.4 & 4.7

The traditional Linux kernel has very low scalability for name lookups in the directory entry cache. This means there is reduced performance in returning information pertaining to a specific file path when there are multiple threads trying to access files in common parent directories due to the kernel serializing the process. This problem is solved in the patched kernel by introducing a new counter to keep track of threads actively looking at the directory entry cache. If a certain thread threatens an entry currently in use by another, the default locking protocol is used to avoid race conditions. If the activities have no bearing on each other, the situation is rightfully ignored, allowing for much faster access to different different entries in the directory entry cache.¹

There are many other locks/mutexes that have special cases where they don't need to lock. Others can be split from locking the whole data structure to locking a part of it. Both these changes remove or reduce bottlenecks.¹

Per-Core Data Structures: Section 4.5

Three centralized data structures were causing bottlenecks due to lock contention - a per-superblock list of open files, vfsmount table, and the packet buffers free list. Each data structure was decentralized to per-core versions of itself. In the case of vfsmount, the central data structure was maintained and any per-core misses got written from the central table to the per-core table.¹

Eliminating false sharing: Section 4.6

Misplaced variables on the cache cause different cores to request the same line, but using different variables. With two different cores requesting the same variable, with one reading and the other writing, there was a severe bottleneck. By moving the often written variable to another line the bottleneck was removed.¹

Conclusion

memcached: Section 5.3

memcached nearly doubles throughput at 48 cores on the paper's implementation over the stock implementation. After improvements to the Linux kernel memcached is limited by hardware. Improvements to hardware scalability, virtual queue handling in this case, will allow further improvements to memcached.¹

Apache: Section 5.4

Apache keeps fairly a equal amount of throughput up to 36 cores. Then it slopes downwards. At 48 cores it still has an improved throughput of more than 12 times. Apache like memcached is limited by hardware. At higher core counts, the network card simply cannot handle the number of packets, and the FIFO queue it holds for them overflows.¹

gmake: Section 5.6

gmakes run time is nearly unchanged by the implementation changes presented in this paper. This is largely because the program has serial sections of code and some processes that finish somewhat later then all others, which prevents perfect scalability. Even with these limitations, gmake achieves the greatest level of scalability out of the three programs (35 times speed up for 48 cores.)¹

No reason to give up traditional kernel design

The contribution of this paper is a lot of research that has focus upon techniques and methods for scalability. This is accomplished through programming of applications alongside kernel programming. This research contributes by evaluating the scalability discrepancies of applications programming and kernel programming. Key discoveries in this research show the effectiveness of the kernel in handling scaling amongst CPU cores. In looking at the issue of scalability it is important to note the factors which hinder scalability.

It has been shown that simple scaling techniques can be effective in increasing scalability. The authors looked at three different approaches to removing the bottlenecks within the system. The first was to see if there were issues within the linux kernel application, the second was to identify issues with the application design and the third was to address how the application interacts with the linux kernel services. Through this approach, the authors were able to quickly identify problems, such as bottlenecks, and apply simple techniques in fixing the issues at hand to reap some beneficial aspects. Some of the sections listed provide insight into the improvements that can be reaped from these optimizations.

Through this research on various techniques, as listed above, it was determined by the authors that the Linux kernel itself has many incorporated techniques used to improve scalability. The authors actually go on to speculate that "perhaps it is the case that Linux's system-call API is well suited to an implementation that avoids unnecessary contention over kernel objects."¹ This tends to show that the work in the Linux community has improved Linux a large amount and is current with modern techniques for optimization.

The kernel design does not seem to affect the scalability of the system. While kernel changes do help, they are not enough; the applications must also be scalable. Finally, there must also be improvements to currently non-scalable hardware.

Critique

Aside from a few acronyms that were unexplained (e.g. Linux TLB), the paper has no real stylistic problems.

memcached: Section 5.3

memcached is treated with near perfect fairness in the paper. Its an in-memory service, so the ignored storage I/O bottleneck does not affect it at all. Likewise the "stock" and "PK" implementations are given the same test suite, so there is no advantage given to either. memcached itself is non-scalable, so the MIT team was forced to run one instance per-core to keep up throughput. The FAQ at memcached.org's wiki suggests using multiple implementations per-server as a work around to another problem, which implies that there is no great problem with running multiple servers on one machine.⁷

Apache: Section 5.4

Linux has a built in kernel flaw where network packets are forced to travel though multiple queues before they arrive at queue where they can be processed by the application. This imposes significant costs on multi-core systems due to the cost associated with queue locking. This flaw inherently diminishes the performance of Apache on multi-core systems due to multiple threads spread across cores being forced to deal with these mutex (mutual exclusion) costs. For the sake of this experiment, Apache had a separate instance on every core listening on different ports, which is not a practical real world application, but merely an attempt to implement better parallel execution on a traditional kernel. The patched kernel implementation of the network stack is also specific to the problem at hand, which is processing multiple short lived connections across multiple cores. Although this provides a performance increase in the given scenario, in more general applications network performance might suffer. These tests were also rigged to avoid bottlenecks in place by network and file storage hardware. Meaning, making the proposed modifications to the kernel won't necessarily produce the same increase in productivity as described in the article. This is very much evident in the tests where performance degrades past 36 cores due to limitation of the networking hardware.¹

gmake: Section 5.6

Since the inherent nature of gmake makes it quite parallel, the testing and updating that were attempted on gmake resulted in essentially the same scalability results for both the stock and modified kernel. The only change that was found was that gmake spent slightly less time at the system level because of the changes that were made to the system's caching. As stated in the paper, the execution time of gmake relies quite heavily on the compiler that is used with gmake, so depending on which compiler was chosen, gmake could run worse or even slightly better. In any case, there seems to be no fairness concerns when it comes to the scalability testing of gmake as the same application load-out was used for all of the tests.¹

Conclusion: Sections 6 & 7

Each testing case has the same specific parameters, with the only changes being from the stock kernel to the PK kernel. The assumption that held true for all tests was that the I/O bottleneck was irrelevant to the tests. Thus the conclusion has no concerns with respect to the content of the paper.

References

¹Silas Boyd-Wickizer, Austin T. Clements, Yandong Mao, Aleksey Pesterev, M. Frans Kaashoek, Robert Morris, and Nickolai Zeldovich. An Analysis of Linux Scalability to Many Cores. MIT CSAIL, 2010. http://www.usenix.org/events/osdi10/tech/full_papers/Boyd-Wickizer.pdf

²J. Kuskin, D. Ofelt, M. Heinrich, J. Heinlein, R. Simoni, K. Gharachorloo, J. Chapin, D. Nakahira, J. Baxter, M. Horowitz, A. Gupta, M. Rosenblum, and J. Hennessy. The Stanford FLASH multiprocessor. In Proc. of the 21st ISCA, pages 302–313,1994.

³P. E. McKenney, D. Sarma, A. Arcangeli, A. Kleen, O. Krieger, and R. Russell. Read-copy-update. In Proceedings of the Linux Symposium 2002, pages 338-367, Ottawa Ontario, June 2002

⁴C. Yan, Y. Chen, and S. Yuanchun. OSMark: A benchmark suite for understanding parallel scalability of operating systems on large scale multi-cores. In 2009 2nd International Conference on Computer Science and Information Technology, pages 313–317, 2009

⁵GNU.org. GNU Make Manual. [Online] http://www.gnu.org/software/make/manual/

⁶MOSBENCH [Online] http://pdos.csail.mit.edu/mosbench/

⁷memcached wiki [Online] memcached's wiki: http://code.google.com/p/memcached/wiki/FAQ#Can_I_use_different_size_caches_across_servers_and_will_memcache

⁸J. Corbet. Receive Packet Steering, November 2009. http://lwn.net/Articles/362339/.

⁹J. Edge. Receive Flow Steering, April 2010. http://lwn.net/Articles/382428/.