Lustre (file system)

From Wikipedia, the free encyclopedia

Lustre

Developer:	Cluster File Systems
Latest release:	1.4.7 / August 25, 2006
OS:	Linux
Use:	Distributed file system
License:	GPL
Website:	http://www.lustre.org

Lustre is a Free Software distributed file system, generally used for large scale cluster computing. The name is a merge of Linux and clusters. The project aims to provide a file system to cope with clusters of tens of thousands of nodes with petabytes of storage capacity, without compromising on speed or security, and is available under the GNU GPL.

Cluster File Systems are the designers, developers, and maintainers of Lustre with input from many other individuals and companies.

Many of the fastest supercomputers in the world are clusters using the Lustre file system for storage, such as systems at ORNL, PNNL, LLNL and LANL.

Each file stored on a Lustre file system is considered as an object. Lustre presents all clients a standard POSIX semantics and concurrent read and write access to the shared objects. A Lustre file system has four functional units. These are an Meta data server (MDS) for storing the meta data; an Object storage target (OST) for storing the actual data; an Object storage server (OSS) for managing the OSTs; client(s) for accessing and the actual usage of the data. OSTs are block-based devices. An MDS, OSS, and an OST can be on the same node or on different nodes. Lustre does not directly talk or manage OSTs, and delegates this responsibility to OSSs to ensure scalability for large-scale clusters and supercomputers.

On a Massively Parallel Processor (MPP), computational processors can access Lustre file system as redirecting their I/O requests to the job launcher service node if that one is configured as a Lustre client. Although it is the easiest method, it provides poor overall performance. A slightly more complicated way of providing a very good overall performance is to use liblustre library. Liblustre is a user-level Lustre library allowing computational processors to mount and use the Lustre file system as a client, bypassing the redirection back to the service node. Using liblustre, the computational processors can access a Lustre file system, even if the service node on which the job was launched is not a Lustre client. Liblustre allows data movement directly between application space and the Lustre OSSs without the need for an intervening data copy through the lightweight kernel, thus providing low latency, high bandwidth access from computational processors to the Lustre file system directly. Due to its impressive performance characteristics and its scalability, it is the most suitable solution utilizing Lustre over MPP systems. Liblustre is the biggest design difference between Lustre on MPPs such as Cray XT3 and on conventional clustered computational systems.

Lustre technology has been integrated in the HP StorageWorks Scalable File Share product.