NTFS

NTFS
Developer(s) Microsoft
Full name New Technology File System[1]
Introduced July 1993 with Windows NT 3.1
Partition identifier 0x07 (MBR)
EBD0A0A2-B9E5-4433-87C0-68B6B72699C7 (GPT)
Structures
Directory contents B+ tree[2]
File allocation Bitmap
Bad blocks $BadClus (MFT Record)
Limits
Max. volume size 264 clusters − 1 cluster (format);
256 TiB − 64 KiB (implementation)[3]
Max. file size 16 EiB – 1 KiB (format);
16 TiB – 64 KiB (Windows 7, Windows Server 2008 R2 or earlier implementation)[3] 256 TiB – 64 KiB (Windows 8, Windows Server 2012 implementation)[4]
Max. number of files 4,294,967,295 (232-1)[3]
Max. filename length 255 UTF-16 code units[5]
Allowed characters in filenames In POSIX namespace, any UTF-16 code unit (case-sensitive) except U+0000 (NUL) and / (slash). In Win32 namespace, any UTF-16 code unit (case-insensitive) except U+0000 (NUL) / (slash) \ (backslash) : (colon) * (asterisk) ? (Question mark) " (quote) < (less than) > (greater than) and | (pipe)[5]
Features
Dates recorded Creation, modification, POSIX change, access
Date range 1 January 1601 – 28 May 60056 (File times are 64-bit numbers counting 100-nanosecond intervals (ten million per second) since 1601, which is 58,000+ years)
Date resolution 100 ns
Forks Yes (see § Alternate data streams below)
Attributes Read-only, hidden, system, archive, not content indexed, off-line, temporary, compressed
File system permissions ACLs
Transparent compression Per-file, LZ77 (Windows NT 3.51 onward)
Transparent encryption Per-file,
DESX (Windows 2000 onward),
Triple DES (Windows XP onward),
AES (Windows XP Service Pack 1, Windows Server 2003 onward)
Data deduplication Yes (Windows Server 2012)[6]
Other
Supported operating systems Windows NT 3.1 and later
OS X 10.3 and later (read-only)
Linux kernel version 2.2 and later
ReactOS (read-only)

NTFS (New Technology File System[1]) is a proprietary file system developed by Microsoft.[1] Starting with Windows NT 3.1, it is the default file system of Windows NT family.[7]

NTFS has several technical improvements over FAT and HPFS (High Performance File System), the file systems that it superseded, such as improved support for metadata, and the use of advanced data structures to improve performance, reliability, and disk space utilization, plus additional extensions, such as security access control lists (ACL) and file system journaling.

History

In the mid-1980s, Microsoft and IBM formed a joint project to create the next generation of graphical operating system. The result of the project was OS/2, but Microsoft and IBM disagreed on many important issues and eventually separated: OS/2 remained an IBM project and Microsoft worked on Windows NT.

The OS/2 file system HPFS contained several important new features. When Microsoft created their new operating system, they borrowed many of these concepts for NTFS.[8] Probably as a result of this common ancestry, HPFS and NTFS use the same disk partition identification type code (07).

Using the same Partition ID Record Number is unusual, since there were dozens of available codes, and other major file systems have their own code. FAT has more than nine (one each for FAT12, FAT16, FAT32, etc.). Algorithms identifying the file system in a partition type 07 must perform additional checks.

Developers

NTFS developers include:[9]

Versions

Microsoft has released five versions of NTFS:

The NTFS.sys version number (e.g. v5.0 in Windows 2000) should not be confused with the NTFS format version number (v3.1 since Windows XP).[12]

Although subsequent versions of Windows added new file system-related features, they did not change NTFS itself. For example, Windows Vista implemented NTFS symbolic links, Transactional NTFS, partition shrinking and self-healing.[13] All except NTFS symbolic links are operating system's features.

Features

NTFS v3.0 includes several new features over its predecessors: sparse file support, disk usage quotas, reparse points, distributed link tracking, and file-level encryption, also known as the Encrypting File System (EFS).

Scalability

In theory, the maximum NTFS volume size is 264−1 clusters. However, the maximum NTFS volume size as implemented in Windows XP Professional is 232−1 clusters partly due to partition table limitations. For example, using 64 kB clusters, the maximum Windows XP NTFS volume size is 256 TB minus 64 KB. Using the default cluster size of 4 kB, the maximum NTFS volume size is 16 TB minus 4 kB. (Both of these are vastly higher than the 128 GB limit lifted in Windows XP SP1.) Because partition tables on master boot record (MBR) disks only support partition sizes up to 2 TB, dynamic or GPT volumes must be used to create NTFS volumes over 2 TB. Booting from a GPT volume to a Windows environment in a Microsoft supported way requires a system with UEFI and 64-bit support.[14]

The maximum theoretical file size on NTFS is 16 EB (16 × 10246 or 264 bytes) minus 1 kB or 18,446,744,073,709,550,592 bytes. With Windows 8 and Windows Server 2012, the maximum file size implemented is 256 TB minus 64 KB or 281,474,976,645,120 bytes.[4]

NTFS supports a maximum cluster size of 64 kB.[15]

Journaling

NTFS is a journaling file system and uses the NTFS Log ($LogFile) to record metadata changes to the volume. It is a critical functionality of NTFS (a feature that FAT does not provide) for ensuring that its internal complex data structures (notably the volume allocation bitmap), or data moves performed by the defragmentation API, the modifications to MFT records (such as moves of some variable-length attributes stored in MFT records and attribute lists), and indices (for directories and security descriptors) will remain consistent in case of system crashes, and allow easy rollback of uncommitted changes to these critical data structures when the volume is remounted.

The USN Journal (Update Sequence Number Journal) is a system management feature that records (in $Extend\$UsnJrnl) changes to files, streams and directories on the volume, as well as their various attributes and security settings. The journal is made available for applications to track changes to the volume.[16] This journal can be enabled or disabled on non-system volumes.[17]

Hard links

Hard links allows different file names to directly refer to the same file contents.

Hard links are similar to directory junctions, but refer to files instead. Hard links may link to files in the same volume only because each volume has its own MFT. Hard links have their own file metadata, so a change in file size or attributes under one hard link may not update the others until they are opened.[18]

Hard links were originally included to support the POSIX subsystem in Windows NT.[19]

Windows uses hard links to support short (8.3) filenames in NTFS. Operating system support is needed because there are legacy applications that can work only with 8.3 filenames. In this case, an additional filename record and directory entry is added, but both 8.3 and long file name are linked and updated together, unlike a regular hard link.

The NTFS file system has a limit of 1024 hard links on a file.[20]

Alternate data streams (ADS)

Alternate data streams allow more than one data stream to be associated with a filename, using the format "filename:streamname" (e.g., "text.txt:extrastream").

NTFS Streams were introduced in Windows NT 3.1, to enable Services for Macintosh (SFM) to store resource forks. Although current versions of Windows Server no longer include SFM, third-party Apple Filing Protocol (AFP) products (such as GroupLogic's ExtremeZ-IP) still use this feature of the file system. Very small ADS (called Zone.Identifier) are added by Internet Explorer and recently by other browsers to mark files downloaded from external sites as possibly unsafe to run; the local shell would then require user confirmation before opening them.[21] When the user indicates that they no longer want this confirmation dialog, this ADS is deleted.

Alternate streams are not listed in Windows Explorer, and their size is not included in the file's size. They are ignored when the file is copied or moved to another file system without ADS support, attached to an e-mail, or uploaded to a website. Thus, using alternate streams for critical data may cause problems. Microsoft provides a tool called Streams[22] to view streams on a selected volume. Starting with Windows PowerShell 3.0, it is possible to manage ADS natively with seven cmdlets: Add-Content, Clear-Content, Get-Content, Get-Item, Out-String, Remove-Item, Set-Content.[23]

Malware has used alternate data streams to hide code.[24] As a result, malware scanners and other special tools now check for alternate data streams.

File compression

NTFS can compress files using LZNT1 algorithm (a variant of the LZ77[25]). Files are compressed in 16-cluster chunks. With 4 kB clusters, files are compressed in 64 kB chunks. The compression algorithms in NTFS are designed to support cluster sizes of up to 4 kB. When the cluster size is greater than 4 kB on an NTFS volume, NTFS compression is not available.[26] If the compression reduces 64 kB of data to 60 kB or less, NTFS treats the unneeded 4 kB pages like empty sparse file clusters—they are not written. This allows for reasonable random-access times as the OS just has to follow the chain of fragments. However, large compressible files become highly fragmented since every chunk < 64KB becomes a fragment.[27][28] According to Microsoft's NTFS Development team’s research, a file over 50-60 GB is not a “reasonable size” to attempt to compress it, on an NTFS volume with a 4KB (default) cluster (block) size, and this maximum “reasonable size” goes down sharply for volumes with smaller cluster sizes.[29] Single-user systems with limited hard disk space can benefit from NTFS compression for small files, from 4 kB to 64 kB or more, depending on compressibility. Files less than 900 bytes or so are stored within the directory entry at the MFT.[30]

Flash memory, such as SSD drives do not have the head movement delays of hard disk drives, so fragmentation has only small effects. Users of fast multi-core processors will find improvements in application speed by compressing their applications and data as well as a reduction in space used.[31] Note that SSDs with Sandforce controllers already compress data. However, since less data is transferred, there is a reduction in I/Os.

The best use of compression is for files that are repetitive, seldom written, usually accessed sequentially, and not themselves compressed. Log files are an ideal example.

Compressing system files needed at boot time, like drivers, NTLDR, winload.exe, or BOOTMGR may prevent the system from booting correctly, as compression filters are not available then.[32] However, in later editions of Windows, compression of important system files is disallowed.

Files may be compressed or decompressed individually (via changing the advanced attributes) for a drive, directory, or directory tree, becoming a default for the files inside.

Although read–write access to compressed files is mostly[33] transparent, Microsoft recommends avoiding compression on server systems and/or network shares holding roaming profiles because it puts a considerable load on the processor.[34] Since many fragments are created for compressible files, defragmentation may take longer.

Sparse files

A sparse file: Empty bytes don't need to be saved, thus they can be represented by metadata.

Sparse files are files interspersed with empty segments for which no actual storage space is used. To the applications, the file looks like an ordinary file with empty regions seen as regions filled with zeros.[35]

Database applications, for instance, may use sparse files.[36] As with compressed files, the actual sizes of sparse files are not taken into account when determining quota limits.[37]

Volume Shadow Copy

The Volume Shadow Copy Service (VSS) keeps historical versions of files and folders on NTFS volumes by copying old, newly overwritten data to shadow copy via copy-on-write technique. The user may later request an earlier version to be recovered. This also allows data backup programs to archive files currently in use by the file system. On heavily loaded systems, Microsoft recommends setting up a shadow copy volume on a separate disk.[38]

Windows Vista also introduced persistent shadow copies for use with System Restore and Previous Versions features. Persistent shadow copies, however, are deleted when an older operating system mounts that NTFS volume. This happens because the older operating system does not understand the newer format of persistent shadow copies.[39]

Transactions

As of Windows Vista, applications can use Transactional NTFS (TxF) to group changes to files together into a transaction. The transaction will guarantee that all changes happen, or none of them do, and it will guarantee that applications outside the transaction will not see the changes until they are committed.[40]

It uses similar techniques as those used for Volume Shadow Copies (i.e. copy-on-write) to ensure that overwritten data can be safely rolled back, and a CLFS log to mark the transactions that have still not been committed, or those that have been committed but still not fully applied (in case of system crash during a commit by one of the participants).

Transactional NTFS does not restrict transactions to just the local NTFS volume, but also includes other transactional data or operations in other locations such as data stored in separate volumes, the local registry, or SQL databases, or the current states of system services or remote services. These transactions are coordinated network-wide with all participants using a specific service, the DTC, to ensure that all participants will receive same commit state, and to transport the changes that have been validated by any participant (so that the others can invalidate their local caches for old data or rollback their ongoing uncommitted changes). Transactional NTFS allows, for example, the creation of network-wide consistent distributed file systems, including with their local live or offline caches.

Microsoft now advise against using TxF: "Microsoft strongly recommends developers utilize alternative means..." since "TxF may not be available in future versions of Microsoft Windows"[41]

Security

In NTFS, each file or folder is assigned a security descriptor that defines its owner and contains two access control lists (ACLs). The first ACL, called discretionary access control list (DACL), defines exactly what type of interaction (e.g. reading, writing, executing or deleting) is allowed or forbidden by which user or groups of users. For example, files in C:\Program Files folder may be read and executed by all users but only modified by a user holding administrative privileges.[42] Windows Vista adds mandatory access control info to DACLs. DACLs are the primary focus of User Account Control in Windows Vista and later.

The second ACL, called system access control list (SACL), defines which interactions with the file or folder are to be audited and whether they should be logged when the activity is successful, failed or both. For example, auditing can be enabled on sensitive files of a company, so that its managers get to know when someone tries to delete them or make a copy of them, and whether he or she succeeds.[42]

Encryption

Encrypting File System (EFS) provides strong[43] and user-transparent encryption of any file or folder on an NTFS volume. EFS works in conjunction with the EFS service, Microsoft's CryptoAPI and the EFS File System Run-Time Library (FSRTL). EFS works by encrypting a file with a bulk symmetric key (also known as the File Encryption Key, or FEK), which is used because it takes a relatively small amount of time to encrypt and decrypt large amounts of data than if an asymmetric key cipher is used. The symmetric key that is used to encrypt the file is then encrypted with a public key that is associated with the user who encrypted the file, and this encrypted data is stored in an alternate data stream of the encrypted file. To decrypt the file, the file system uses the private key of the user to decrypt the symmetric key that is stored in the file header. It then uses the symmetric key to decrypt the file. Because this is done at the file system level, it is transparent to the user.[44] Also, in case of a user losing access to their key, support for additional decryption keys has been built into the EFS system, so that a recovery agent can still access the files if needed. NTFS-provided encryption and NTFS-provided compression are mutually exclusive; however, NTFS can be used for one and a third-party tool for the other.

The support of EFS is not available in Basic, Home and MediaCenter versions of Windows, and must be activated after installation of Professional, Ultimate and Server versions of Windows or by using enterprise deployment tools within Windows domains.

Quotas

Disk quotas were introduced in NTFS v3. They allow the administrator of a computer that runs a version of Windows that supports NTFS to set a threshold of disk space that users may use. It also allows administrators to keep track of how much disk space each user is using. An administrator may specify a certain level of disk space that a user may use before they receive a warning, and then deny access to the user once they hit their upper limit of space. Disk quotas do not take into account NTFS's transparent file-compression, should this be enabled. Applications that query the amount of free space will also see the amount of free space left to the user who has a quota applied to them.

Reparse points

Main article: NTFS reparse point

NTFS reparse points, introduced in NTFS v3, are used by associating a reparse tag in the user space attribute of a file or directory. Microsoft includes several default tags including NTFS symbolic links, directory junction points and volume mount points. When the object manager (see Windows NT line executive) parses a file system name lookup and encounters a reparse attribute, it will reparse the name lookup, passing the user controlled reparse data to every file system filter driver that is loaded into Windows. Each filter driver examines the reparse data to see whether it is associated with that reparse point, and if that filter driver determines a match, then it intercepts the file system call and executes its special functionality.

Resizing

Starting with Windows Vista Microsoft added the built-in ability to shrink or expand a partition, but this capability is limited because it will not relocate page file fragments or files that have been marked as unmovable. So shrinking will often require relocating or disabling any page file, the index of Windows Search, and any Shadow Copy used by System Restore. Various third-party tools are capable of resizing NTFS partitions.

Internals

NTFS file system permissions on a Windows Vista system

Internally, NTFS uses B+ trees to index file system data. Although complex to implement, this allows faster file look up times in most cases. A file system journal is used to guarantee the integrity of the file system metadata but not individual files' content. Systems using NTFS are known to have improved reliability compared to FAT file systems.[45]

NTFS allows any sequence of 16-bit values for name encoding (file names, stream names, index names, etc.) except 0x0000. This means UTF-16 code units are supported, but the file system does not check whether a sequence is valid UTF-16 (it allows any sequence of short values, not restricted to those in the Unicode standard). File names are limited to 255 UTF-16 code units. Certain names are reserved in the volume root directory and cannot be used for files. These are $MFT, $MFTMirr, $LogFile, $Volume, $AttrDef, . (dot), $Bitmap, $Boot, $BadClus, $Secure, $UpCase, and $Extend.[3] (dot) and $Extend are both directories; the others are files. The NT kernel limits full paths to 32,767 UTF-16 code units. There are some additional restrictions on code points and file names.[46]

Partition Boot Sector

NTFS Boot Sector contents
Byte Offset Field length Typical Value Field Name Purpose
0x00 3 bytes 0xEB5290 JMP instruction Causes execution to continue after the data structures in this boot sector.
0x03 8 bytes "NTFS    "
Word "NTFS" followed by four trailing spaces (0x20)		 OEM ID This is the magic cookie that indicates this is an NTFS file system.
0x0B 2 bytes 0x0200 Bytes per sector How many bytes are in a sector?
0x0D 1 byte 0x08 Sectors Per Cluster How many sectors are in a cluster?
0x0E 2 bytes 0x0000 Reserved Sectors How much space is reserved by the OS. Not specified as to where in the reference cited.
0x10 3 bytes 0x000000 Unused This field is always 0
0x13 2 bytes 0x0000 Unused by NTFS This field is always 0
0x15 1 byte 0xF8 Media Descriptor Not specified in reference cited.
0x16 2 bytes 0x0000 Unused This field is always 0
0x18 2 bytes 0x003F Sectors Per Track How many sectors are there per track?
0x1A 2 bytes 0x00FF Number Of Heads How many magnetic read-write heads are there in this drive?
0x1C 4 bytes 0x0000003F Hidden Sectors How many hidden sectors are there? Location not specified in reference cited.
0x20 4 bytes 0x00000000 Unused Not used by NTFS
0x24 4 bytes 0x80008000 Unused Not used by NTFS
0x28 8 bytes 0x00000000007FF54A Total sectors How many sectors are in this partition?
0x30 8 bytes 0x0000000000000004 $MFT cluster number Which cluster contains the $MFT?
0x38 8 bytes 0x000000000007FF54 $MFTMirr cluster number Which cluster contains the mirror (backup) $MFT
0x40 1 byte 0xF6 Bytes Per File Record Segment How many bytes per file record segment? If negative then value is 2 to the power of -value, e.g. if value is 0xF6 then record length is 2^10 (10==-0xf6)
0x44 1 byte 0x01 Clusters Per Index Buffer How many clusters per index buffer? Algorithm as for bytes per record above
0x45 3 bytes 0x000000 Unused This field is not used by NTFS
0x48 8 bytes 0x1C741BC9741BA514 Volume Serial Number A unique random number assigned to this partition, to keep things organized.
0x50 4 bytes 0x00000000 Checksum A checksum of the previous values. Algorithm not described in cited article.
0x54 426 bytes Bootstrap Code The code that loads the rest of the operating system. This is pointed to by the first 3 bytes of this sector.
0x01FE 2 bytes 0xAA55 End-of-sector Marker This flag indicates that this is a valid boot sector.

[47] The OS first looks at the 8 bytes at 0x30 to find the cluster number of the $MFT, then multiplies that number by the number of sectors per cluster (1 byte found at 0x0D) and the number of bytes per sector (2 bytes found at 0x0b). This value is the byte offset to the $MFT, which is described below.

Master File Table

In NTFS, all file, directory and metafile data—file name, creation date, access permissions (by the use of access control lists), and size—are stored as metadata in the Master File Table (MFT). This abstract approach allowed easy addition of file system features during Windows NT's development—an example is the addition of fields for indexing used by the Active Directory software. This also enables fast file search software such as Everything, Ultrasearch, or SwiftSearch to locate named local files and folders included in the MFT very fast, without requiring any other index.

The MFT structure supports algorithms which minimize disk fragmentation.[48] A directory entry consists of a filename and a "file ID", which is the record number representing the file in the Master File Table. The file ID also contains a reuse count to detect stale references. While this strongly resembles the W_FID of Files-11, other NTFS structures radically differ.

Two copies of the MFT are stored in case of corruption. If the first record is corrupted, NTFS reads the second record to find the MFT mirror file. Locations for both files are stored in the boot sector.[49]

For protection, in Windows 7, attempting to open the MFT via an Administrator Command Prompt will cause the system to BSoD with an error code of 0x24 (NTFS_FILE_SYSTEM). In Windows 8, running the same command will bring up an Open File dialogue, but all programs will show Access Denied errors. The command to "open" the MFT is cd / and then start $mft. Both commands must be run at the same time.

Metafiles

NTFS contains several files that define and organize the file system. In all respects, most of these files are structured like any other user file ($Volume being the most peculiar), but are not of direct interest to file system clients. These metafiles define files, back up critical file system data, buffer file system changes, manage free space allocation, satisfy BIOS expectations, track bad allocation units, and store security and disk space usage information. All content is in an unnamed data stream, unless otherwise indicated.

List of NTFS metafiles
Segment Number File Name Purpose
0 $MFT Describes all files on the volume, including file names, timestamps, stream names, and lists of cluster numbers where data streams reside, indexes, security identifiers, and file attributes like "read only", "compressed", "encrypted", etc.
1 $MFTMirr Duplicate of the first vital entries of $MFT, usually 4 entries (4 Kilobytes).
2 $LogFile Contains transaction log of file system metadata changes.
3 $Volume Contains information about the volume, namely the volume object identifier, volume label, file system version, and volume flags (mounted, chkdsk requested, requested $LogFile resize, mounted on NT 4, volume serial number updating, structure upgrade request). This data is not stored in a data stream, but in special MFT attributes: If present, a volume object ID is stored in an $OBJECT_ID record; the volume label is stored in a $VOLUME_NAME record, and the remaining volume data is in a $VOLUME_INFORMATION record. Note: volume serial number is stored in file $Boot (below).
4 $AttrDef A table of MFT attributes that associates numeric identifiers with names.
5 . Root directory. Directory data is stored in $INDEX_ROOT and $INDEX_ALLOCATION attributes both named $I30.
6 $Bitmap An array of bit entries: each bit indicates whether its corresponding cluster is used (allocated) or free (available for allocation).
7 $Boot Volume boot record. This file is always located at the first clusters on the volume. It contains bootstrap code (see NTLDR/BOOTMGR) and a BIOS parameter block including a volume serial number and cluster numbers of $MFT and $MFTMirr.
8 $BadClus A file that contains all the clusters marked as having bad sectors. This file simplifies cluster management by the chkdsk utility, both as a place to put newly discovered bad sectors, and for identifying unreferenced clusters. This file contains two data streams, even on volumes with no bad sectors: an unnamed stream contains bad sectors—it is zero length for perfect volumes; the second stream is named $Bad and contains all clusters on the volume not in the first stream.
9 $Secure Access control list database that reduces overhead having many identical ACLs stored with each file, by uniquely storing these ACLs in this database only (contains two indices: $SII (Standard_Information ID) and $SDH (Security Descriptor Hash), which index the stream named $SDS containing actual ACL table).[2]
10 $UpCase A table of unicode uppercase characters for ensuring case-insensitivity in Win32 and DOS namespaces.
11 $Extend A file system directory containing various optional extensions, such as $Quota, $ObjId, $Reparse or $UsnJrnl.
12–23 Reserved for $MFT extension entries. Extension entries are additional MFT records that contain additional attributes that do not fit in the primary record. This could occur if the file is sufficiently fragmented, has many streams, long filenames, complex security, or other rare situations.
24 $Extend\$Quota Holds disk quota information. Contains two index roots, named $O and $Q.
25 $Extend\$ObjId Holds link tracking information. Contains an index root and allocation named $O.
26 $Extend\$Reparse Holds reparse point data (such as symbolic links). Contains an index root and allocation named $R.
27— Beginning of regular file entries.

These metafiles are treated specially by Windows, handled directly by the NTFS.SYS driver and are difficult to directly view: special purpose-built tools are needed.[50] As of Windows 7, the NTFS driver completely prohibits user access, resulting in a BSoD whenever an attempt to execute a metadata file is made. One such tool is the nfi.exe ("NTFS File Sector Information Utility") that is freely distributed as part of the Microsoft "OEM Support Tools". For example, to obtain information on the "$MFT"-Master File Table Segment the following command is used: nfi.exe c:\$MFT[51] Another way to bypass the restriction is to use 7-zip's file manager and go to the low-level NTFS path \\.\X:\ (where X:\ resembles any drive/partition). Here, 3 new folders will appear: $EXTEND, [DELETED] (a pseudo-folder that 7-zip uses to attach files deleted from the file system to view), and [SYSTEM] (another pseudo-folder that contains all the NTFS metadata files). This trick can be used from removable devices (USB flash drives, external hard drives, SD Cards, etc.) inside Windows, but doing this on the active partition requires offline access (namely WinRE)

Attribute lists, attributes, and streams

For each file (or directory) described in the MFT record, there is a linear repository of stream descriptors (also named attributes), packed together in one or more MFT records (containing the so-called attributes list), with extra padding to fill the fixed 1 KB size of every MFT record, and that fully describes the effective streams associated with that file.

Each attribute has an attribute type (a fixed-size integer mapping to an attribute definition in file $AttrDef), an optional attribute name (for example, used as the name for an alternate data stream), and a value, represented in a sequence of bytes. For NTFS, the standard data of files, the alternate data streams, or the index data for directories are stored as attributes.

According to $AttrDef, some attributes can be either resident or non-resident. The $DATA attribute, which contains file data, is such an example. When the attribute is resident (which is represented by a flag), its value is stored directly in the MFT record. Otherwise, clusters are allocated for the data, and the cluster location information is stored as data runs in the attribute.

All attributes of a given file may be displayed by using the nfi.exe ("NTFS File Sector Information Utility") that is freely distributed as part of the Microsoft "OEM Support Tools".[51]

Windows system calls may handle alternate data streams.[3] Depending on the operating system, utility and remote file system, a file transfer might silently strip data streams.[3] A safe way of copying or moving files is to use the BackupRead and BackupWrite system calls, which allow programs to enumerate streams, to verify whether each stream should be written to the destination volume and to knowingly skip unwanted streams.[3]

Resident vs. non-resident attributes

To optimize the storage and reduce the I/O overhead for the very common case of attributes with very small associated value, NTFS prefers to place the value within the attribute itself (if the size of the attribute does not then exceed the maximum size of an MFT record), instead of using the MFT record space to list clusters containing the data; in that case, the attribute will not store the data directly but will just store an allocation map (in the form of data runs) pointing to the actual data stored elsewhere on the volume.[52] When the value can be accessed directly from within the attribute, it is called "resident data" (by computer forensics workers). The amount of data that fits is highly dependent on the file's characteristics, but 700 to 800 bytes is common in single-stream files with non-lengthy filenames and no ACLs.

The allocation map is stored in a form of data runs with compressed encoding. Each data run represents a contiguous group of clusters that store the attribute value. For files on a multi-GB volume, each entry can be encoded as 5 to 7 bytes, which means a 1 KB MFT record can store about 100 such data runs. However, as the $ATTRIBUTE_LIST also has a size limit, it is dangerous to have more than 1 million fragments of a single file on an NTFS volume, which also implies that it is in general not a good idea to use NTFS compression on a file larger than 10 GB.[53]

The NTFS file system driver will sometimes attempt to relocate the data of some of the attributes that can be made non-resident into the clusters, and will also attempt to relocate the data stored in clusters back to the attribute inside the MFT record, based on priority and preferred ordering rules, and size constraints.

Since resident files do not directly occupy clusters ("allocation units"), it is possible for an NTFS volume to contain more files on a volume than there are clusters. For example, a 74.5 GB partition NTFS formats with 19,543,064 clusters of 4 KB. Subtracting system files (a 64 MB log file, a 2,442,888-byte Bitmap file, and about 25 clusters of fixed overhead) leaves 19,526,158 clusters free for files and indices. Since there are four MFT records per cluster, this volume theoretically could hold almost 4 × 19,526,158 = 78,104,632 resident files.

Opportunistic locks

Opportunistic locks (oplocks) allow clients to alter their buffering strategy for a given file or stream in order to increase performance and reduce network use.[54] Oplocks apply to the given open stream of a file and do not affect oplocks on a different stream.

Oplocks can be used to transparently access files in the background. A network client may avoid writing information into a file on a remote server if no other process is accessing the data, or it may buffer read-ahead data if no other process is writing data.

Windows supports four different types of oplocks:

Opportunistic locks have been enhanced in Windows 7 and Windows Server 2008 R2 with per-client oplock keys.[55]

Time

Windows NT and its descendants keep internal timestamps as UTC and make the appropriate conversions for display purposes. Therefore, NTFS timestamps are in UTC.

For historical reasons, the versions of Windows that do not support NTFS all keep time internally as local zone time, and therefore so do all file systems other than NTFS that are supported by current versions of Windows. This means that when files are copied or moved between NTFS and non-NTFS partitions, the OS needs to convert timestamps on the fly. But if some files are moved when daylight saving time (DST) is in effect, and other files are moved when standard time is in effect, there can be some ambiguities in the conversions. As a result, especially shortly after one of the days on which local zone time changes, users may observe that some files have timestamps that are incorrect by one hour. Due to the differences in implementation of DST in different jurisdictions, this can result in a potential timestamp error of up to 4 hours in any given 12 months.[56]

Interoperability

While the different NTFS versions are for the most part fully forward- and backward-compatible, there are technical considerations for mounting newer NTFS volumes in older versions of Microsoft Windows. This affects dual-booting, and external portable hard drives. For example, attempting to use an NTFS partition with "Previous Versions" (a.k.a. Volume Shadow Copy) on an operating system that does not support it will result in the contents of those previous versions being lost.[57] A Windows command-line utility called convert.exe can convert supporting file systems to NTFS, including HPFS (only on Windows NT 3.1, 3.5, and 3.51), FAT16 and FAT32 (on Windows 2000 and later).[58][59]

Mac OS X 10.3 and later include read-only support for NTFS-formatted partitions. The GPL-licensed NTFS-3G also works on Mac OS X through FUSE and allows reading and writing to NTFS partitions. A performance enhanced commercial version, called Tuxera NTFS for Mac,[60] is also available from the NTFS-3G developers. Paragon Software Group sells a read-write driver named NTFS for Mac OS X,[61] which is also included on some models of Seagate hard drives.[62] Native NTFS write support has been discovered in Mac OS X 10.6 and later, but is not activated by default, although workarounds do exist to enable the functionality. However, user reports indicate the functionality is unstable and tends to cause kernel panics, probably the reason why write support has not been enabled or advertised.[63]

Linux kernel versions 2.2.0 and later include the ability to read NTFS partitions; kernel versions 2.6.0 and later contain a driver written by Anton Altaparmakov (University of Cambridge) and Richard Russon which supports file read, overwrite and resize. Three userspace drivers (NTFSMount, NTFS-3G and Captive NTFS, a 'wrapping' driver that uses Windows' own driver, ntfs.sys) exist for NTFS support. They are built on the Filesystem in Userspace (FUSE), a Linux kernel module tasked with bridging userspace and kernel code to save and retrieve data. All three are licensed under the terms of the GNU General Public License (GPL). Due to the complexity of internal NTFS structures, both the built-in 2.6.14 kernel driver and the FUSE drivers disallow changes to the volume that are considered unsafe, to avoid corruption. Two proprietary solutions also exist:

eComStation, and FreeBSD offer read-only NTFS support (there is a beta NTFS driver that allows write/delete for eComStation, but is generally considered unsafe). A free third-party tool for BeOS, which was based on NTFS-3G, allows full NTFS read and write. NTFS-3G also works on Mac OS X, FreeBSD, NetBSD, Solaris, QNX and Haiku, in addition to Linux, through FUSE.[64] A free for personal use read/write driver for MS-DOS called "NTFS4DOS" also exists.[65][66] Ahead Software developed a "NTFSREAD" driver (version 1.200) for DR-DOS 7.0x between 2002 and 2004. It was part of their Nero Burning ROM software. OpenBSD offer native read-only NTFS support by default on i386 and amd64 platforms as of version 4.9 released 1 May 2011.[67] Read/write support through NTFS-3G are possible in OpenBSD -current as of 1 November 2013 (first release is OpenBSD 5.5 on 1 May 2014) as OpenBSD henceforth has its own FUSE implementation[68] and NTFS-3G is available from ports.[69]

See also

References

  1. 1 2 3 "1.1 Glossary". [MS-EFSR]: Encrypting File System Remote (EFSRPC) Protocol. Microsoft. 14 November 2013.
  2. 1 2 3 4 Russinovich, Mark. "Inside Win2K NTFS, Part 1". Microsoft Developer Network. Microsoft. Retrieved 2008-04-18.
  3. 1 2 3 4 5 6 7 "How NTFS Works". Windows Server 2003 Technical Reference. 2003-03-28. Retrieved 2011-09-12.
  4. 1 2 "6 Appendix A: Product Behavior". [MS-FSA]: File System Algorithms. Microsoft. 14 November 2013. Retrieved 2012-09-21.
  5. 1 2 Russon, Richard; Fledel, Yuval. "NTFS Documentation" (PDF). Retrieved 2011-06-26.
  6. ↑ Rick Vanover. "Windows Server 8 data deduplication". Retrieved 2011-12-02.
  7. 1 2 Custer, Helen (1994). Inside the Windows NT File System. Microsoft Press. ISBN 978-1-55615-660-1.
  8. ↑ Kozierok, Charles M. (April 17, 2001). "Overview and History of NTFS". PCGuide.
  9. ↑ Custer, Helen (1994). Inside the Windows NT File System. Microsoft Press. p. vii. ISBN 978-1-55615-660-1.
  10. ↑ "Recovering Windows NT After a Boot Failure on an NTFS Drive". Microsoft. November 1, 2006.
  11. ↑ "Inside Win2K NTFS, Part 1". Microsoft. January 26, 2011.
  12. ↑ "New Capabilities and Features of the NTFS 3.1 File System". Microsoft. December 1, 2007.
  13. ↑ Loveall, John (2006). "Storage improvements in Windows Vista and Windows Server 2008" (PowerPoint). Microsoft. pp. 14–20. Retrieved 2007-09-04.
  14. ↑ http://www.rodsbooks.com/gdisk/booting.html
  15. ↑ "[MS-FSA]: File System Algorithms. Appendix A: Product Behavior". Microsoft. Retrieved 2012-01-10.
  16. ↑ "Change Journals (Windows)". MSDN. Retrieved 2010-04-16.
  17. ↑ "Creating, Modifying, and Deleting a Change Journal (Windows)". MSDN. Retrieved 2010-04-16.
  18. ↑ "Hard Links and Junctions". MSDN. Microsoft. 12 October 2013. Retrieved 21 October 2013.
  19. ↑ "Chapter 29 - POSIX Compatibility". MS Windows NT Workstation 4.0 Resource Guide. Microsoft. 1995. Retrieved 21 October 2013.
  20. ↑ "MSDN - CreateHardLink function". Retrieved 14 January 2016.
  21. ↑ Russinovich, Mark E.; Solomon, David A.; Ionescu, Alex (2009). "File Systems". Windows Internals (5th ed.). Microsoft Press. p. 921. ISBN 978-0-7356-2530-3. One component in Windows that uses multiple data streams is the Attachment Execution Service[...] depending on which zone the file was downloaded from [...] Windows Explorer might warn the user
  22. ↑ Sysinternals Streams v1.56
  23. ↑ "FileSystem Provider". Microsoft. 9 August 2012. Retrieved 23 January 2015.
  24. ↑ Malware utilising Alternate Data Streams?, AusCERT Web Log, 21 August 2007
  25. ↑ "File Compression and Decompression". MSDN Platform SDK: File Systems. Retrieved 2005-08-18.
  26. ↑ "The Default Cluster Size for the NTFS and FAT File Systems". Microsoft. January 31, 2002. Retrieved 2012-01-10.
  27. ↑ "Understanding NTFS Compression". Retrieved 2011-03-16.
  28. ↑ "Shrinking the gap: carving NTFS-compressed files". Retrieved 2011-05-29.
  29. ↑ "Understanding NTFS Compression". Retrieved 2016-02-15.
  30. ↑ "How NTFS Works". 2003-03-28. Retrieved 2011-10-24.
  31. ↑ Masiero, Manuel (2011-12-01). "Should You Compress Data On Your SSD?". Tom's Hardware. Bestofmedia Group. Retrieved 2013-04-05.
  32. ↑ "Disk Concepts and Troubleshooting". Microsoft. Retrieved 2012-03-26.
  33. ↑ "Read-Only Filegroups and Compression". SQL Server 2008 Books Online. Microsoft. November 2009. Retrieved 2010-04-20.
  34. ↑ "Best practices for NTFS compression in Windows." Microsoft Knowledge Base. Retrieved on 2005-08-18.
  35. ↑ "Sparse Files". MSDN. Microsoft. 12 October 2013. Retrieved 21 October 2013.
  36. ↑ Kandoth, Suresh B. (4 March 2009). "Sparse File Errors: 1450 or 665 due to file fragmentation: Fixes and Workarounds". CSS SQL Server Engineers. Microsoft. Retrieved 21 October 2013.
  37. ↑ "Sparse Files and Disk Quotas". MSDN Library. Microsoft. 12 October 2013. Retrieved 21 October 2013.
  38. ↑ "Designing a Shadow Copy Strategy". TechNet Library. Microsoft. 28 March 2003. Retrieved 2008-01-15.
  39. ↑ cfsbloggers (July 14, 2006). "How restore points and other recovery features in Windows Vista are affected when you dual-boot with Windows XP". The Filing Cabinet. Retrieved 2007-03-21.
  40. ↑ "Transactional NTFS". MSDN. Microsoft. Retrieved 2007-02-02.
  41. ↑ "Transactional NTFS (TxF)". Windows Dev Center (MSDN). Microsoft. Retrieved 24 May 2015.
  42. 1 2 "How Security Descriptors and Access Control Lists Work". TechNet. Microsoft. Retrieved 4 September 2015.
  43. ↑ Morello, John (February 2007). "Security Watch Deploying EFS: Part 1". Technet Magazine. Microsoft. Retrieved 2009-01-25.
  44. ↑ "How EFS Works". Windows 2000 Resource Kit. Microsoft. Retrieved 25 February 2014.
  45. ↑ "Chapter 18 - Choosing a File System". MS Windows NT Workstation 4.0 Resource Guide. Microsoft. Retrieved 25 February 2014.
  46. ↑ "Naming Files, Paths, and Namespaces". MSDN. Microsoft. Naming Conventions. Retrieved 25 February 2014.
  47. ↑ NTFS Partition Boot Sector Information on structure of the boot sector.
  48. ↑ "Master File Table". MSDN. July 2, 2012.
  49. ↑ NTFS Master File Table (MFT). Information on structure of MFT file.
  50. ↑ Since Windows XP, it is very difficult to view a listing of these files: they exist in the root directory's index, but the Win32 interface filters them out. In NT 4.0, the command line dir command would list the metafiles in the root directory if /a were specified. In Windows 2000, dir /a stopped working, but dir /a \$MFT worked.
  51. 1 2 "OEM Support Tools Phase 3 Service Release 2 Availability". Microsoft Corporation. 2007-02-21. Retrieved 2010-06-16. Windows NT File System (NTFS) File Sector Information Utility [...] A tool used to dump information about an NTFS volume
  52. ↑ blogs.technet.com: Jeff Hughes, The Four Stages of NTFS File Growth (2009)
  53. ↑ Microsoft KB: A heavily fragmented file in an NTFS volume may not grow beyond a certain size (2012)
  54. ↑ http://msdn.microsoft.com/en-us/library/cc308442.aspx
  55. ↑ http://technet.microsoft.com/en-us/library/ff383236(WS.10).aspx
  56. ↑ "Beating the Daylight Saving Time bug and getting correct file modification times" The Code Project
  57. ↑ cfsbloggers (July 14, 2006). "How restore points and other recovery features in Windows Vista are affected when dual-booting with Windows XP". The Filing Cabinet. Retrieved 2007-03-21.
  58. ↑ "How to Convert FAT Disks to NTFS". Microsoft Corporation. 2001-10-25. Retrieved 2007-08-27.
  59. ↑ "How to use Convert.exe to convert a partition to the NTFS file system". Microsoft Corporation. 2007-02-12. Retrieved 2010-12-26.
  60. ↑ "Tuxera NTFS for Mac". Tuxera. August 30, 2011. Retrieved September 20, 2011.
  61. ↑ "NTFS for Mac OS X, communication channel between Mac OS X and Windows". Paragon Software Group. Retrieved September 20, 2011.
  62. ↑ Seagate Read/Write NTFS driver for Mac OS X
  63. ↑ Alvares, Milind (October 2, 2009). "Snow Leopard's hidden NTFS read/write support". Retrieved 18 September 2010.
  64. ↑ "NTFS-3G Stable Read/Write Driver". 2009-07-25.
  65. ↑ Avira NTFS4DOS Personal at the Wayback Machine (archived June 19, 2010)
  66. ↑ Download of NTFS4DOS.EXE NTFS driver for DOS at Softpedia.com
  67. ↑ http://openbsd.com/49.html
  68. ↑ "OpenBSD adds fuse(4) support for adding file systems in userland". OpenBSD Journal. 2013-11-08. Retrieved 2013-11-08.
  69. ↑ "ntfs_3g-2014.2.15 – FUSE NTFS driver with read/write support". OpenBSD ports. 2014-01-05. Retrieved 2015-02-14.

Further reading

  • Bolosky, William J.; Corbin, Scott; Goebel, David; Douceur, John R. (2 December 2008). "Single Instance Storage in Windows 2000" (PDF). Microsoft. 
  • Custer, Helen (1994). Inside the Windows NT File System. Microsoft Press. ISBN 978-1-55615-660-1. 
  • Nagar, Rajeev (1997). Windows NT File System Internals: A Developer's Guide. O'Reilly. ISBN 978-1-56592-249-5. 

External links

This article is issued from Wikipedia - version of the Tuesday, February 16, 2016. The text is available under the Creative Commons Attribution/Share Alike but additional terms may apply for the media files.