Data Execution Prevention

Data Execution Prevention (DEP) is a security feature included in modern operating systems. It marks areas of memory as either "executable" or "nonexecutable", and allows only data in an "executable" area to be run by programs, services, device drivers, etc. It is known to be available in OS X, Microsoft Windows and iOS^[1] operating systems.

DEP protects against some program errors, and helps prevent certain malicious exploits, especially attacks that store executable instructions in a data area via a buffer overflow.^[2] It does not protect against attacks that do not rely on execution of instructions in the data area. Other security features such as address space layout randomization, structured exception handler overwrite protection (SEHOP) and Mandatory Integrity Control, can be used in conjunction with DEP.^[3]

DEP runs in two modes: hardware-enforced DEP for CPUs that can mark memory pages as nonexecutable, and software-enforced DEP with limited protection for CPUs that do not have hardware support. Software-enforced DEP does not protect against execution of code in data pages, but counters SEH overwrite, another type of attack.

DEP was introduced on Windows in 2004 with Windows XP Service Pack 2,^[4] while Apple introduced DEP when they moved to x86 in 2006.^[2]

Mode of enforcement

Hardware enforcement

Hardware-enforced DEP enables the NX bit on compatible CPUs, through the automatic use of PAE kernel in 32-bit Windows and the native support on 64-bit kernels. Windows Vista DEP works by marking certain parts of memory as being intended to hold only data, which the NX or XD bit enabled processor then understands as non-executable.^[4] This helps prevent buffer overflow attacks from succeeding. In Windows, from version Vista, whether DEP is enabled or disabled for a particular process can be viewed on the Processes tab in the Windows Task Manager.

Some legitimate programs erroneously but harmlessly execute instructions in a data area; enabling Data Execution Prevention can have the unintended consequence of causing such software to malfunction. This needs to be remediated by the software developer; until then the program can be flagged as being allowed to execute code in data memory, although this is vulnerable to a possible targeted attack.

Microsoft Windows

If an x86 processor supports DEP, and the system's BIOS supports it and it has been enabled, either by the manufacturer or the user,^[5] then the NX features are turned on in Windows, on a limited "OptIn" basis.^[6] (PAE must also be enabled in 32-bit Windows.) This setting provides protection only for a limited set of Windows system and binary files. To achieve full protection, the user must choose either "OptOut" ("turn on DEP for all program services except those I select"), covering all programs and processes not specifically exempted, or "AlwaysOn", covering all. These are configurable through the System Properties interface.^[7] If DEP is not supported by the particular x86 processor used, no protection is given. Intel's IA-64 architecture also has a form of NX, supported by Windows.

Software enforcement

One kind of software DEP (without the use of the NX bit) is Microsoft's "Safe Structured Exception Handling" (SafeSEH). For properly compiled applications, SafeSEH checks that, when an exception is raised during program execution, the exception's handler is one defined by the application as it was originally compiled. The effect of this protection is that an attacker is not able to add his own exception handler which he has stored in a data page through unchecked program input. ^[4][8]

Limitations

Where code is written and executed at runtime—a JIT compiler is a prominent example—the compiler can potentially be used to produce exploit code (e.g. using JIT Spray) that has been flagged for execution and therefore would not be trapped by DEP.^[9]

Issues

Compatibility

DEP occasionally encounters software problems, usually with older software that was not compiled and tested to conform to its restrictions. Users have experienced problems using various command-line commands that are a part of Microsoft's Services for Unix, which is included as part of Vista as well as Windows Server 2003 R2.

These problems may be prevented by disabling DEP, but this increases vulnerability of the system to malware. DEP can be turned off on a per-application basis, or turned off entirely for all non-essential Windows programs and services.^[10] Microsoft recommends that DEP not be globally disabled where an application malfunctions due to incompatibility with DEP. Instead, the supplier of the offending software should be contacted for an updated version that does not violate DEP; until the problem is corrected DEP may be disabled on an exception basis for the offending application only.^[11]

DEP is applied to an entire process, so even an application compatible with DEP may need to have it disabled if a non-DEP-compliant extension is added that runs in the same process space. For example, DEP-related problems can occasionally occur with DEP-compliant core operating system components such as Windows Explorer, Internet Explorer and Windows Installer as they support in-process third party extensions or plugins that may not be DEP-compliant.^[12]

See also

• Executable space protection
• Heap overflow
• Stack buffer overflow
• Stack-smashing protection

References

External links

• A description of DEP and how to configure it
• Guide to hardware-based DEP protection
• The "Understanding Data Execution Prevention" help topic incorrectly states the default setting for DEP in Windows Server 2003 Service Pack 1
• Easy steps to define DEP exceptions in Vista and XP