Distributed File System Security

COMP 5102 - Distributed Operating Systems

Trevor Gelowsky

March 1st, 2011

Abstract

This paper provides an overview on how data security is maintained both in traditional distributed file systems, and cloud-based systems. It begins by providing background information on distributed file systems, and then continues with a series of examples of how the data is secured. Finally, an overview of a new system providing not only data, but computational security is discussed.

An Introduction to Distributed File System Security

Distributed file systems are a core part of modern distributed operating systems. Without them, it would be impossible to catalogue, sort, or access the huge volumes of data required by distributed systems conveniently. Similarly, they facilitate high-performance computing (HPC) in a way not possible before the advent of distributed file systems.

There are numerous examples of distributed file system in use today. These include the Google File System (GFS) [1]⁠, Lustre[2]⁠, Parallel Virtual File System Version 2 (PVFS2) [3]⁠, and The Global File System (GFS) [4]⁠, Kosmos File System (KFS), and Hadoop[5]⁠. Each of these are designed to provide high-availability, and scalability.

With the advent of distributed file systems, however, there is the added issue of security. It is no longer the case where it is necessarily possible for all data to exist within a walled garden. Now it is possible for a single file system to be spread among multiple cloud or grid computing services (such as Amazon S3, among many others). This has necessitated the creation higher levels of abstraction which are capable of operating on a much larger scale, and with a higher level of security than traditional distributed file systems.

So how do modern distributed filing systems provide guarantees of data security, and what mechanisms of authentication are used to safeguard data? To answer these questions, this paper will explore this evolving field of study, drawing in examples from existing solutions.

The Scope of This Paper

This literature review will be focusing on the various aspects of distributed filing systems, and how they address the core issue of security. Background material will be provided for the sake of clarity where required, but it will be assumed that the basic underlying concepts of distributed file systems are well understood.

This paper will begin with a discussion of the basic commercially available distributed file systems, and their built-in features, and then continue with the common systems built on top of these to provide or enhance the security features of the underlying distributed filing system.

Like the field of distributed file systems itself, the area of security is quite diverse, and for that reason this paper will focus on a few key examples highlighting the variety, and promenade types of solutions available for solving the problems of distributed file system security, and distributed computation in general.

Distributed File System Security

Distributed file system security can be provided in two ways. The first is to have some kind of an authentication and security layer built-in to the file system itself. Examples of these include Lustre [2]⁠, Panasas, Parallel Virtual File System Version 2 (PVFS2), and the Redhat Global File System (RGFS) [6]⁠. These systems typically consider security among their primary concerns, and therefore take great care to provide a mechanism to authenticate and protect access to data.

Alternatively, many distributed filing systems, such as the Google File System (GFS), Kosmos File System (KFS), and Hadoop, posses no built-in security mechanism[6]⁠. It is assumed in these cases that the nodes and clients in the system are trusted and secure. To extend the functionality of these systems a number of solutions have been developed in order to create a trusted, and secure computing environment based on these insecure distributed filing systems.

File Systems with a Dedicated Security Mechanism

The vast majority of distributed file systems that include built-in security mechanisms rely on a trusted storage-area network (SAN) with UNIX-like permissions (user/group) being used to manage the access of data via one or more coordinating nodes [3,4,6]⁠. These systems rely on the trusted nodes in the network to provide access to data in the way they best see fit.

There are two notable exceptions, however, which provides authentication in a much finer-grained way using a more advanced request authentication system: The Andrew File System (AFS), and The Lustre file system, and its descendants.

Network File System

The Network File System protocol (NFS) [7]⁠ was introduced as a way of providing distributed file system services. Until version 4, however, security was not a concern, and it followed a largely UNIX-based security scheme that was user and group centric. As of NFS version 4, however, it became more heavily influenced by alternative systems that were now available (including the Andrew File System below) and introduced a more comprehensive security scheme [8]⁠. In addition, many security layers have been created that use NFS as a back-end for low-level storage[9]⁠.

The Andrew File System

The Andrew File System (AFS) [10]⁠ behaves a lot like it's predecessor, NFS, except in that it uses a series of Access Control Lists (ACL) [11]⁠ to govern access to files. It does this by employing the Kerberos authentication mechanism [12]⁠ to apply a fine-grained security properties to data being stored in the distributed file system. This system, however, does not mandate the use of system-wide encryption.

The Lustre Distributed File System

The Lustre file system bases its security around the popular Public Key Infrastructure (PKI)[2]⁠, and Kerberos [12]⁠. This is similar to the AFS, and allows for fine-grained security profiles to be applied to every piece of data stored in the system. Like many distributed file systems, it divides information into to categories: Data, and Metadata. Data is stored in Lustre Object Storage Targets (OST), while metadata is stored in Metadata Servers (MDS).

In order to provide security, Lustre grants access to data in a four-step process. Fist the user requests information from the Metadata Server (MDS). Second, the MDS authenticates the request using the Public Key Infrastructure (PKI). Next, the MDS transmits data to the OST required, and the client granting access to the information. Finally, the OST and the client connect directly with the OST validating the request before sending data to the client.

This system effectively prevents replay attacks, data interception, and client identity theft. Since all communication between nodes in this system is encrypted, data security is guaranteed.

Cloud-Based and Other Security Mechanisms

Given the lack of built-in support for securing data in many common distributed filing system, it's unsurprising that higher-level systems have been developed in order to provide data security. These systems tend to be based around cloud storage [13]⁠. Cloud storage differs from traditional distributed file systems in that from the perspective of the system there is no hardware concerns, and it is expected that the provider of the cloud storage will provide all the required replication, reliability, and backup. Cloud storage does not traditionally provide faculties to manage security.

Provable Data Possession and Proof of Retrievability

Provable Data Possession (PDP) [14]⁠ is a scheme used to verify that the underlying storage mechanism possesses the original copy of the data stored on it, without modification, by storing a set of metadata separately in a client store. This metadata utilizes a homomorphic verifiable tag generated from a probabilistic sampling of the resource in question. This scheme means that tampering of the underlying file can be detected.

A fully homomorphic tag ensures that it can be recomputed without needing to reveal the encryption key itself [15]⁠. In a PDP system, this allows the tags computed for multiple file blocks can be combined into one value. This value can then be challenged randomly by clients at any time without having to actually retrieve the block of data in question.

Proof of Retrievability (POR) [16]⁠ builds on the idea of PDP by adding spot-checking and error-correcting codes that guarantee both possession and retrievability of the data stored on the underlying file system [17]⁠.

Attribute-Based Encryption

There is a relatively new style of security being introduced into modern distributed file systems known as Attribute-Based Encryption (ABE) [18]⁠. These systems, unlike more central systems that rely on some kind of an ACL, are identity-based and encrypt objects based on the attributes required to access those objects. The required cryptographic keys are maintained and managed separately. In addition, these schemes are collision-resistant. This solved the problems of distributed and cloud storage by ensuring that the underlying storage mechanism does not ever have access to the unencrypted data.

To phrase it in another way, if only administrators should be able to access a given resource, then the administrator attribute would become a part of the encryption key for the resource, so you would require at least that attribute to access the information. Users lacking that attribute would be unable to access the data.

SecCloud: Secure Storage and Computation

This system takes things one step further than just securing the data on the storage-end of things. SecCloud is a system that attempts to address both the concerns in secure cloud storage, and secure computation [17]⁠. It accomplished this by incorporating many of the same concepts as PDP and POR, extending them with an auditing scheme based on a probabilistic sampling technique to further increase overall data security.

This is extended by adding in a hash tree based commit scheme. Once the commit is completed, the result is then challenged, and verified ensuring not only the completion of the commit, but also verifies that the request was completed[17]⁠.

Conclusion

In this paper the central issues surrounding the security of distributed filing systems were discussed. As with most non-trivial areas of computer science, the correct solution for any problem will vary greatly based on the individual requirements that any system is required to meet. For many systems, it may be perfectly acceptable to assume that no security is adequate, whereas with others a great deal of time and resources may be allocated to ensuring that the data being stored in these petabyte-scaled systems remains secure.

References

[1] S. Ghemawat, H. Gobioff, and S.-T. Leung, “The Google file system,” ACM SIGOPS Operating Systems Review, vol. 37, Dec. 2003, p. 29.

[2] S.-qin Liu, X.-sheng Li, J. Shuo, J. Wang, and H.-hui Liu, Lustre Security Mechanism: Models, Schemes and Research Based on PKI, IEEE, 2010.

[3] J.M. Kunkel and T. Ludwig, “Performance Evaluation of the PVFS2 Architecture,” 15th EUROMICRO International Conference on Parallel, Distributed and Network-Based Processing (PDPʼ07), Feb. 2007, pp. 509-516.

[4] S. Soltis, G. Erickson, K. Preslan, and T. Ruwart, “The Global File System: A file system for shared disk storage,” IEEE Transactions on Parallel and Distributed Systems, vol. 1, 1997, p. 1.

[5] J. Shafer, S. Rixner, and A.L. Cox, “The Hadoop distributed filesystem: Balancing portability and performance,” Performance Analysis of Systems & Software (ISPASS), 2010 IEEE International Symposium on, IEEE, 2010, p. 122–133.

[6] T.D. Thanh, S. Mohan, E. Choi, S. Kim, and P. Kim, “A Taxonomy and Survey on Distributed File Systems,” 2008 Fourth International Conference on Networked Computing and Advanced Information Management, Sep. 2008, pp. 144-149.

[7] B. Callaghan, B. Pawlowski, and P. Staubach, “RFC 1813: NFS version 3 protocol specification, June 1995,” See also RFC1094 [Sun89]. Status: Informational, 1995, pp. 1-127.

[8] B. Callaghan, D. Robinson, and R. Thurlow, “Network File System (NFS) version 4 Protocol,” 2003, pp. 1-276.

[9] R. Pletka and C. Cachin, “Cryptographic Security for a High-Performance Distributed File System,” 24th IEEE Conference on Mass Storage Systems and Technologies (MSST 2007), Sep. 2007, pp. 227-232.

[10] R. Tobbicke, “Distributed file systems: focus on Andrew File System/Distributed File Service (AFS/DFS),” Proceedings Thirteenth IEEE Symposium on Mass Storage Systems. Toward Distributed Storage and Data Management Systems, 1994, pp. 23-26.

[11] S.V. Nagaraj, “Access control in distributed object systems: problems with access control lists,” Enabling Technologies: Infrastructure for Collaborative Enterprises, 2001. WET ICE 2001. Proceedings. Tenth IEEE International Workshops on, IEEE, 2002, p. 163–164.

[12] B.C. Neuman and T. Tsʼo, “Kerberos: an authentication service for computer networks,” IEEE Communications Magazine, vol. 32, 1994, pp. 33-38.

[13] J. Wu, L. Ping, X. Ge, Y. Wang, and J. Fu, “Cloud Storage as the Infrastructure of Cloud Computing,” 2010 International Conference on Intelligent Computing and Cognitive Informatics, Jun. 2010, pp. 380-383.

[14] G. Ateniese, R. Burns, R. Curtmola, J. Herring, L. Kissner, Z. Peterson, and D. Song, “Provable data possession at untrusted stores,” Proceedings of the 14th ACM conference on Computer and communications security, New York, New York, USA: ACM, 2007, p. 598–609.

[15] C. Gentry, “A fully homomorphic encryption scheme,” Stanford University, 2009.

[16] K.D. Bowers, A. Juels, and A. Oprea, “Proofs of retrievability: Theory and implementation,” Proceedings of the 2009 ACM workshop on Cloud computing security, ACM, 2009, p. 43–54.

[17] L. Wei, H. Zhu, Z. Cao, and W. Jia, “SecCloud: Bridging Secure Storage and Computation in Cloud,” 2010 IEEE 30th International Conference on Distributed Computing Systems Workshops, Jun. 2010, p. xxxix-xl.

[18] M. Pirretti, P. Traynor, P. McDaniel, and B. Waters, “Secure attribute-based systems,” Proceedings of the 13th ACM conference on Computer and communications security - CCS ’06, 2006, p. 99.