COMP 3000 Essay 2 2010 Question 7

Paper

Ad Hoc Synchronization Considered Harmful

Weiwei Xiong University of California, San Diego

Soyeon Park, Jiaqi Zhang, Yuanyuan Zhou University of Illinois at Urbana-Champaign

Zhiqiang Ma Intel

Research Problem

As the computer industry continues to shift towards multicore processors, concurrent programming and the use of multithreaded designs has increased to keep up with this growing trend. Multithreaded applications can be found in a variety of popular applications today as they take advantage of the multithreaded approach. However, the concepts behind concurrent programming bring with them a host of potential dangers in the form of race conditions and deadlocks resulting from bad programming design and threads accessing shared memory. Fortunately, there are well known and standard methods for dealing with these problems, i.e synchronization primitives. But in real world situations, due to a variety of reasons, as we shall see, programmers often implement their own "ad hoc" synchronizations that eschew common design standards. Ad hoc synchronizations are not well documented and are not discovered by traditional tools for race conditions that look for standard synchronization primitives.

The paper we are discussing addresses these concerns in two regards, first it details a thorough study of ad hoc synchronizations. It details their nature, dangers, impact on bug detection tools and prevalence in several major open-source applications. Secondly, it introduces SyncFinder, a program that detects all ad hoc synchronizations and automatically annotates the source code where ad hoc sychnronizations are found. This can see use in conjunction with other data race checkers to improve accuracy and to build custom tools for finding deadlocks and bad programming practices.

With detailed analysis of ad hoc synchronization and study of their occurrences in several applications, the research ultimately concludes that they are harmful and should be removed. At the same time, SyncFinder detects and documents ad hoc synchronizations in the source code enabling programmers for the first time to easily track and remove them.

Background Concepts

Concurrent Programming

Concurrent programming is a style of programming where multiple threads of execution run concurrently to perform a single task. The thread of execution share a number of resources. Particularly in multi-core processor system and in distributed environment this style of programming can result in significant performance gains. One significant challenge of concurrent programming is coordinating the different threads of execution, this is usually done using synchronization primitives.

Synchronization Primitives

Synchronization primitives represent some of the basic tools offered by the system containing the concurrent program to facilitate coordination between of threads of execution. They are generally used to synchronize between threads, and to protect shared resource⁸. Some type of synchronization primitives common to many are locks, mutexes, semaphores, and monitors. Locks are really a superset of synchronization primitives since mutexes, semaphores, and monitors are all locks. Additionally there are read/write locks that only lock when a reader obtains the lock. Latches a type of lock that unlocks when a specified number of threads have obtained it which is very useful in facilitate threads all getting to known state, there are a myriad of other locks. Mutexes are mutually exclusive locks that threads employ to lock a resource that they need preventing other. No other threads can access them at that point. Once they are finished, they release the lock and the other threads can then lock and access the resource. Monitors are a type of mutex that contain a condition variables which is a variable that if not true, releases the lock and blocks the thread until the certain condition is met, by another thread changing value of the variable. This allows the original thread to only continue execute when it is safe to perform its operation. Synchronization primitives can be misused and lead to a host of other problems generally referred to collectively as race conditions.

Race conditions, deadlocks

Race conditions are an unintended side-effects of programming in concurrent systems, they occur when two or more processes have access to a shared resource and at least one of them have write privilege. This leads to processes modifying the data that all processes share as other may be reading them and results in the reading of stale/incorrect data. They will occur during the execution of the program and often times very difficult to detect or manipulate data in subtle ways.

Deadlock is when two or more processes share a resource and each process is waiting on the other processes to unlock the resource. It becomes a circular chain and no process can continue.

Both these issues occur in concurrent programming and although there are no general solutions for deadlock⁹, there are suitable methods for dealing with them, and in the case of race conditions, using mutual exclusion locks and synchronization primitives can prevent race conditions. But no programmer is infallible and so there is always the issue of race conditions and deadlocks present in production code.

Ad Hoc Synchronization

Ad hoc synchronizations are loops called sync loops that continue until certain conditions are met via outside variables called sync variables. They are designed to control the flow of thread execution much like locking and unlocking of resources. There can be multiple sync variables in a sync loop and they can have multiple exit conditions and dependencies. The diversity of the sync loops, their dependencies and execution paths leads to difficulty in finding them.

Contribution

With concurrent programming commonly used in modern applications, we face many issues that result from having simultaneous execution. In order to maintain a concurrent system, synchronization is required to ensure that the executing tasks do not interfere with each other avoiding potential race conditions. However, many programmers do not use proper synchronization primitives to deal with these issues. Rather, they implement synchronizations in an ad hoc fashion. The paper we are discussing shows that ad hoc synchronizations, though implemented as a solution to concurrency issues, are indeed undesirable in a system. This paper details the characteristics of ad hoc synchronizations and the issues associated with this programming construct and introduces the program, SyncFinder, which is used to identify such synchronizations in code.

Findings

In order to identify the characteristics of ad hoc synchronizations, 12 mainstream programs were examined to find instances of ad hoc synchronizations. These programs were either of server, desktop or scientific type, including Apache, MySQL and Mozilla. Through manual inspection of the source code, these characteristics of ad hoc synchronizations were found.

1. In all programs studied, it was found that each had numerous ad hoc synchronizations implemented. The number of synchronizations found ranged from 6-83, with server type programs inhabiting the higher portion of the interval. It is likely that programmers use this type of synchronization for two reasons.

• In order to ensure a certain order of execution in the case of a concurrent system, programmers will use ad hoc synchronization to superimpose this order. With traditional synchronization techniques, this can vary between systems. As the order can vary, it is difficult to create a common interface.
• Some synchronization techniques introduce heavy-weight synchronization primitives. As such, programmers will use ad hoc synchronizations to avoid this and supposedly protect performance.

2. Often, it is very hard to identify an ad hoc loop as a synchronization method. They are hard to distinguish from other computational loops and as the implementations are diverse, it is hard to pinpoint them from the code. This makes the system hard to maintain, as other programmers will not be able to identify ad hoc loops implemented by another and debugging programs cannot recognize them as issues.

3. It was found that ad hoc synchronizations often introduce bugs into the system such as deadlocks or hangs. As these are different than those caused by locks and other synchronizations it is hard for detection tools to recognize them if they were not first identified either manually or automatically.

4. As they are not easily recognizable, it is hard for bug detection tools to fix issues presented by ad hoc synchronizations. In fact, it is often the case that these tools either do not find these issues or report them as false positives as the tool is unaware of the "work arounds" put into affect by using ad hoc synchronization. Since they cannot find these problems, it severely impacts the effectiveness of such tools. This also impacts analysis of performance. Synchronization is quite costly and if a tool cannot recognize the formm of synchronization, a false report is generated and the programmer will not be aware. This may cause poor decisions on the part of the programmer just from the fact that ad hoc synchronizations are hard to identify.

5. The reason ad hoc synchronizations are hard to identify stems from the fact that there is no single way of implementing it. The ways in which ad hoc synchronizations are done are quite diverse and so it is hard to identify just on a few criteria. Some typical characteristics of an ad hoc synchronization follow.

• These loops can contain one or multiple exit conditions. Some or all of these exit conditions may be satisfied by remote threads while others may be satisfied locally.
• Often, exit conditions depend on sync variables, variables that are shared with other tasks
• In some cases, the synchronization does not wait idly and rather performs other computations while checking the sync variable periodically

Despite the dangers of using ad hoc synchronization, programmers continue to use this method. It is found that, in comments, programmers have stated that possibly their implementations are unsafe but proceed to use ad hoc synchronization techniques. The reasoning behind these decisions have already been outlined in point 1. A better practice of synchronization would be to replace ad hoc synchronizations with synchronization primitives, primitives already present in standard POSIX thread libraries. However, it is often difficult to replace ad hoc synchronizations with synchronization primitives and doing this may not fulfill the concerns presented in point 1.

SyncFinder

SyncFinder is a tool built and designed by the authors of the paper for the purpose of identifying and annotating instances of ad hoc synchronization in concurrent programs built in C or C++. The main goal of this was to aid programmers in better structuring their code, while simultaneously allowing for other tools to be utilized, recognizing them as synchronizations. It has demonstrated itself to be very effective in this area where other similar tools have failed, as it analyzes the code in a unique way that specifically tracks down sync loops that implement ad hoc synchronization.

How it works

There are two possibilities to consider when searching for ad hoc synchronizations. You can either analyze runtime traces via a dynamic method, or analyze the source code in a static method. Both methods carry with them a number of pros and cons. While a dynamic process is generally more accurate than a static method, it tends to accrue a very large runtime overhead. In addition to this, the dynamic method is somewhat limited in which ad hoc synchronizations it can find by the code coverage of the test cases. Taking these factors into consideration, the authors of the paper opted to pursue a static solution for achieving the goals set out for SyncFinder. SyncFinder uses the LLVM compiler infrastructure.

1. Find Loops

An important commonality between all ad hoc synchronizations is that they are all caused by loops, be they “for”, “while” or “go to” loops. These are generally referred to as "sync loops". The first step in identifying sync loops is The LLVM infrastructure is used to obtain the loop info from the source including a representation of the exit conditions.

2. Identify Sync Loops

The next and most important step is to differentiate between sync loops used for ad hoc synchronization and regular computational loops. It does this by going through the following steps:

• Exit Dependent Variable (EDV) Analysis: EDVs are variables that affect the exit conditions of a loop. A sync variable is a variable related to the synchronization of concurrent programs. Therefore, by identifying any EDVs as sync variables, it can be concluded that the loop is a sync loop.
• Pruning Computational Loops: If a loop has at least one sync condition, it is considered a sync loop. Otherwise, it is pruned out as a computational loop.
• Pruning Condvar Loops: condvar loops are not considered sync loops. SyncFinder will go through all loop candidates and prune out any that make a calls cond_wait inside the loop.

3. Synchronization Pairing

The next step is to find the remote update that would release the sync loop. SyncFinder first finds all write instructions that would modify the sync variables. It then decides if the value that the write assigns to the sync variable would satisfy the exit condition. All those that do not are pruned. SyncFinder also prunes pairings that do not execute concurrently. This is done conservatively due to the limitations of static analysis.

4. SyncFinder Annotation

Finally, SyncFinder annotates these ad hoc synchronizations in a specific way so that other tools are able to find them.

Uses

SyncFinder is a robust tool that can be utilized in a variety of applications such as bug detection, performance profiling and concurrency testing. Using its auto-annotation feature, it is capable of identifying sections of code that demonstrate bad programming practices, which could in turn cause issues such as deadlocks. In addition to this, the authors of the paper were able to expand upon the existing data race detector “Valgrind” in order to take advantage of the annotation system introduced by SyncFinder. Through this, they were able to reduce the number of false positives flagged by the former, while being able to make use of the information it provides.

Accuracy

SyncFinder was tested against 25 concurrent programs that are used across a broad cross-section of application. In testing SyncFinder was able to positively identify 96% of ad hoc synchronizations within the tested programs. False positives were at a rate of only 6%. In further tests, they were able to utilize SyncFinder’s auto-annotation systems to locate and mark 5 deadlocks and 16 potential issues within Apache, MySQL and Mozilla, that had previously been missed by other analysis tools.

Related Work and similar tools

There has been attempts to remove synchronization issues entirely from concurrent programming, such as transactional memory¹, a lock-free synchronization that does not require mutexes, and avoids having to use lock, unlock operations. Other attempts have been made to remove bugs that would otherwise be safe from data races but are are still at risk of unintended effects from thread interactions, such as Atomizer², a dynamic atomicity checker.

There are tools that detect data races such as CHESS³, a dynamic data race checker that runs through all possible thread execution paths and CTrigger⁴, a tool that checks for atomicity violations. The problem with these programs is that they only look for standard synchronization methods and structures, such as lock() and cond_wait(). They are not looking for ad hoc synchronizations.

A similar tool to SyncFinder exists that can detect simple spinning, also an ad hoc synchronization⁵, but it only detects simple spinning and not the more complicated ad hoc variations.

Several studies on bug characteristics⁶ and concurrency bugs⁷ have been composed. This paper complements these studies to better understand the nature of ad hoc synchronizations and their occurrence in concurrent programs.

Critique

1. Uses static approach, dynamic would be better

• As stated in the paper, dynamic introduces run-time overhead and is not guaranteed to find if not executed in the test cases

Dynamic would potentially make it language agnostic.

2. not entirely accurate, some false positives.

3. In terms of style, lots of unnecessary repetition

This paper successfully identifies a new type programming construct ad hoc synchronization. The paper then refers to a body of data that the authors have created that both provides proof of the criteria to identify the construct and illustrates it's frequency and likelihood to introduce bugs. Previous to SyncFinder, debugging tools have for the most part failed to detect ad hoc synchronizations effectively. Because of this, SyncFinder has been shown to be an important tool for future development of software.

As far as shortcomings go, it has a 96% success rate, with 6% false positives. The 6% false positive rate is more or less unavoidable, so it is hard to fault the tool for that, especially with such a high success rate. Even with a 6% false positive rate, the developer still only needs to look through a select few loops to determine which are actually ad hoc synchronizations as opposed to a whole code base.

Finally the tool that the authors have created has been used on ubiquitous applications¹⁰ like Apache and exposed previously unreported bugs. This stands as a testament to both it's effectiveness and validity.

References

¹ M Herlihy and J.E.B. Moss, 2NA0. Transactional Memory: Architectural Support for Lock-Free Data Structures. [online] Available at: <http://www.cs.brown.edu/~mph/HerlihyM93/herlihy93transactional.pdf> [Accessed 23 November 2010].

² C Flanagan and S N Freund, 2NA0. Atomizer: A Dynamic Atomicity Checker For Multithreaded Programs (Summary). [online] Company(optional) Available at: <http://www.cs.williams.edu/~freund/papers/atomizer-padtad.pdf> [Accessed 23 November 2010].

³ T Ball,M Musuvathi and S Qadeer, 2NA0. CHESS: A Systematic Testing Tool for Concurrent. [online] Company(optional) Available at: <http://research.microsoft.com/pubs/70509/tr-2007-149.pdf> [Accessed 23 November 2010].

⁴ Author, 2NA0. Title. [online] Company(optional) Available at: <http://pages.cs.wisc.edu/~shanlu/paper/asplos092-zhou.pdf> [Accessed 23 November 2010].

⁵ LI, T., LEBECK, A. R., AND SORIN, D. J. Spin detection hardware for improved management of multithreaded systems. IEEE Transactions on Parallel and Distributed Systems PDS-17, 6 (June 2006), 508–521.

⁶ Author, 2NA0. Title. [online] Company(optional) Available at: <http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.121.1203&rep=rep1&type=pdf> [Accessed 23 November 2010].

⁷ Author, 2NA0. Title. [online] Company(optional) Available at: <[1]> [Accessed 23 November 2010].

⁸ Author, 2NA0. Title. [online] Company(optional) Available at: <[2]> [Accessed 23 November 2010].

⁹ Author, 2010. Title. [online] Company(optional) Available at: <http://homeostasis.scs.carleton.ca/wiki/index.php/Basic_Synchronization_Principles> [Accessed 23 November 2010].

¹⁰ Author, 2010. Title. [online] Company(optional) Available at: <http://trends.builtwith.com/Web-Server/Apache> [Accessed 30 November 2010].