x86 calling conventions

This article describes the calling conventions used when programming x86 architecture microprocessors.

Calling conventions describe the interface of called code:

• The order in which atomic (scalar) parameters, or individual parts of a complex parameter, are allocated
• How parameters are passed (pushed on the stack, placed in registers, or a mix of both)
• Which registers the callee must preserve for the caller
• How the task of preparing the stack for, and restoring after, a function call is divided between the caller and the callee

This is intimately related with the assignment of sizes and formats to programming-language types. Another closely related topic is name mangling, which determines how symbol names in the code map to symbol names used by the linker. Calling conventions, type representations, and name mangling are all part of what is known as an application binary interface (ABI).

There are often subtle differences in how various compilers implement these conventions, so it is often difficult to interface code which is compiled by different compilers. On the other hand, conventions which are used as an API standard (such as stdcall) are very uniformly implemented.

Historical background

Prior to microcomputers, the machine manufacturer generally provided an operating system and compilers for several programming languages. The calling convention(s) for each platform were those defined by the manufacturer's programming tools.

Early microcomputers before the Commodore Pet and Apple II generally came without an OS or compilers, as did the IBM PC. The only hardware standard for IBM PC-compatible machines was defined by the Intel processors (8086, 80386) and the literal hardware IBM shipped. Hardware extensions and all software standards (save for a BIOS calling convention) were thrown open to market competition.

A multitude of independent software firms offered operating systems, compilers for many programming languages, and applications. Many different calling schemes were implemented by the firms, often mutually exclusive, based on different requirements, historical practices, and programmer creativity.

After the IBM-compatible market shakeout, Microsoft operating systems and programming tools (with differing conventions) predominated, while second-tier firms like Borland and Novell, and open-source projects like GCC, still maintained their own standards. Provisions for interoperability between vendors and products were eventually adopted, simplifying the problem of choosing a viable convention.^[1]

Caller clean-up

In these conventions, the caller cleans the arguments from the stack.

cdecl

The cdecl (which stands for C declaration) is a calling convention that originates from the C programming language and is used by many C compilers for the x86 architecture.^[1] In cdecl, subroutine arguments are passed on the stack. Integer values and memory addresses are returned in the EAX register, floating point values in the ST0 x87 register. Registers EAX, ECX, and EDX are caller-saved, and the rest are callee-saved. The x87 floating point registers ST0 to ST7 must be empty (popped or freed) when calling a new function, and ST1 to ST7 must be empty on exiting a function. ST0 must also be empty when not used for returning a value.

In the context of the C programming language, function arguments are pushed on the stack in the reverse order. In Linux, GCC sets the de facto standard for calling conventions. Since GCC version 4.5, the stack must be aligned to a 16-byte boundary when calling a function (previous versions only required a 4-byte alignment.)^[2]

Consider the following C source code snippet:

int callee(int, int, int);

int caller(void)
{
	int ret;

	ret = callee(1, 2, 3);
	ret += 5;
	return ret;
}

On x86, it will produce the following assembly code (Intel syntax):

caller:
	; make new call frame
	push    ebp
	mov     ebp, esp
	; push call arguments
	push    3
	push    2
	push    1
	; call subroutine 'callee'
	call    callee
	; remove arguments from frame
	add     esp, 12
	; use subroutine result
	add     eax, 5
	; restore old call frame
	pop     ebp
	; return
	ret

The caller cleans the stack after the function call returns.

There are some variations in the interpretation of cdecl,^[3] particularly in how to return values. As a result, x86 programs compiled for different operating system platforms and/or by different compilers can be incompatible, even if they both use the "cdecl" convention and do not call out to the underlying environment. Some compilers return simple data structures with a length of 2 registers or less in the register pair EAX:EDX, and larger structures and class objects requiring special treatment by the exception handler (e.g., a defined constructor, destructor, or assignment) are returned in memory. To pass "in memory", the caller allocates memory and passes a pointer to it as a hidden first parameter; the callee populates the memory and returns the pointer, popping the hidden pointer when returning.

In Linux/GCC, double/floating point values should be pushed on the stack via the x87 pseudo-stack. Like so:

	sub     esp, 8          ; make room for the double
	fld     [ebp + x]       ; load our double onto the floating point stack
	fstp    [esp]           ; push our double onto the stack
	call    funct
	add     esp, 8

Using this method ensures it is pushed on the stack in the correct format.

The cdecl calling convention is usually the default calling convention for x86 C compilers, although many compilers provide options to automatically change the calling conventions used. To manually define a function to be cdecl, some support the following syntax:

void _cdecl funct();

The _cdecl modifier must be included in the function prototype and in the function declaration, to override any other settings that might be in place.

syscall

This is similar to cdecl in that arguments are pushed right-to-left. EAX, ECX, and EDX are not preserved. The size of the parameter list in doublewords is passed in AL.

Syscall is the standard calling convention for 32 bit OS/2 API.

optlink

Arguments are pushed right-to-left. The three lexically first (leftmost) arguments are passed in EAX, EDX, and ECX and up to four floating-point arguments are passed in ST(0) through ST(3), although space for them is reserved in the argument list on the stack. Results are returned in EAX or ST(0). Registers EBP, EBX, ESI, and EDI are preserved.

Optlink is used by the IBM VisualAge compilers.

Callee clean-up

In these conventions, the callee cleans up the arguments from the stack. Functions which utilize these conventions are easy to recognize in ASM code because they will unwind the stack after returning. The x86 ret instruction allows an optional 16-bit parameter that specifies the number of stack bytes to release after returning to the caller. Such code looks like this:

 ret 12

Conventions entitled fastcall or register have not been standardized, and have been implemented differently, depending on the compiler vendor.^[1] Typically register based calling conventions pass one or more arguments in registers which reduces the number of memory accesses required for the call and thus make them usually faster.

pascal

Based on the Borland Pascal programming language's calling convention, the parameters are pushed on the stack in left-to-right order (opposite of cdecl), and the callee is responsible for balancing the stack before return.

This calling convention was common in the following 16-bit APIs: OS/2 1.x, Microsoft Windows 3.x, and Borland Delphi version 1.x. Modern versions of the Windows API use stdcall, which still has the callee restoring the stack as in the Pascal convention, but the parameters are now pushed right to left.

stdcall

The stdcall^[4] calling convention is a variation on the Pascal calling convention in which the callee is responsible for cleaning up the stack, but the parameters are pushed onto the stack in right-to-left order, as in the _cdecl calling convention. Registers EAX, ECX, and EDX are designated for use within the function. Return values are stored in the EAX register.

stdcall is the standard calling convention for the Microsoft Win32 API and for Open Watcom C++.

Microsoft fastcall

Microsoft^[5] or GCC^[6] __fastcall convention (aka __msfastcall) passes the first two arguments (evaluated left to right) that fit into ECX and EDX. Remaining arguments are pushed onto the stack from right to left. When MS compiler compiles for IA64 or AMD64, it ignores the __fastcall keyword and uses the one 64-bit calling convention instead.

Microsoft vectorcall

In Visual Studio 2013, Microsoft introduced the __vectorcall calling convention in response to efficiency concerns from game, graphic, video/audio, and codec developers.^[7] For IA-32 and x64 code, __vectorcall is similar to __fastcall and the original x64 calling conventions respectively, but extends them to support passing vector arguments using SIMD registers. For x64, when any of the first six arguments are vector types (float, double, __m128, __m256, etc.), they are passed in via the corresponding XMM/YMM registers. Similarly for IA-32, up to six XMM/YMM registers are allocated sequentially for vector type arguments from left to right regardless of position. Additionally, __vectorcall adds support for passing homogeneous vector aggregate (HVA) values, which are composite types consisting solely of up to four identical vector types, using the same six registers. Once the registers have been allocated for vector type arguments, the unused registers are allocated to HVA arguments from left to right regardless of position. Resulting vector type and HVA values are returned using the first four XMM/YMM registers.^[8]

Borland register

Evaluating arguments from left to right, it passes three arguments via EAX, EDX, ECX. Remaining arguments are pushed onto the stack, also left to right.^[9] It is the default calling convention of the 32-bit compiler of Delphi, where it is known as register. Some versions of Linux kernel use this convention on i386.^[10]. This calling convention is also used by Embarcadero's C++Builder, where it is called __fastcall.^[11] In this compiler, Microsoft's fastcall can be used as __msfastcall^[12].

Watcom register

Watcom does not support the __fastcall keyword except to alias it to null. The register calling convention may be selected by command line switch. (However, IDA uses __fastcall anyway for uniformity.)

Up to 4 registers are assigned to arguments in the order eax, edx, ebx, ecx. Arguments are assigned to registers from left to right. If any argument cannot be assigned to a register (say it is too large) it, and all subsequent arguments, are assigned to the stack. Arguments assigned to the stack are pushed from right to left. Names are mangled by adding a suffixed underscore.

Variadic functions fall back to the Watcom stack based calling convention.

The Watcom C/C++ compiler also uses the #pragma aux^[13] directive that allows the user to specify their own calling convention. As its manual states, "Very few users are likely to need this method, but if it is needed, it can be a lifesaver".

TopSpeed / Clarion / JPI

The first four integer parameters are passed in registers eax, ebx, ecx and edx. Floating point parameters are passed on the floating point stack – registers st0, st1, st2, st3, st4, st5 and st6. Structure parameters are always passed on the stack. Additional parameters are passed on the stack after registers are exhausted. Integer values are returned in eax, pointers in edx and floating point types in st0.

safecall

In Delphi and Free Pascal on Microsoft Windows, the safecall calling convention encapsulates COM (Component Object Model) error handling, thus exceptions aren't leaked out to the caller, but are reported in the HRESULT return value, as required by COM/OLE. When calling a safecall function from Delphi code, Delphi also automatically checks the returned HRESULT and raises an exception if necessary.

The safecall calling convention is the same as the stdcall calling convention, except that exceptions are passed back to the caller in EAX as a HResult (instead of in FS:[0]), while the function result is passed by reference on the stack as though it were a final "out" parameter. When calling a Delphi function from Delphi this calling convention will appear just like any other calling convention, because although exceptions are passed back in EAX, they are automatically converted back to proper exceptions by the caller. When using COM objects created in other languages, the HResults will be automatically raised as exceptions, and the result for Get functions is in the result rather than a parameter. When creating COM objects in Delphi with safecall, there is no need to worry about HResults, as exceptions can be raised as normal but will be seen as HResults in other languages.

function function_name(a: DWORD): DWORD; safecall;

Returns a result and raises exceptions like a normal Delphi function, but it passes values and exceptions as though it was:

function function_name(a: DWORD; out Result: DWORD): HResult; stdcall;

Either caller or callee clean-up

thiscall

This calling convention is used for calling C++ non-static member functions. There are two primary versions of thiscall used depending on the compiler and whether or not the function uses a variable number of arguments.

For the GCC compiler, thiscall is almost identical to cdecl: The caller cleans the stack, and the parameters are passed in right-to-left order. The difference is the addition of the this pointer, which is pushed onto the stack last, as if it were the first parameter in the function prototype.

On the Microsoft Visual C++ compiler, the this pointer is passed in ECX and it is the callee that cleans the stack, mirroring the stdcall convention used in C for this compiler and in Windows API functions. When functions use a variable number of arguments, it is the caller that cleans the stack (cf. cdecl).

The thiscall calling convention can only be explicitly specified on Microsoft Visual C++ 2005 and later. On any other compiler thiscall is not a keyword. (However, disassemblers, such as IDA, must specify it. So IDA uses keyword __thiscall for this.)

Register preservation

Another part of a calling convention is which registers are guaranteed to retain their values after a subroutine call. According to the Intel ABI to which the vast majority of compilers conform, the EAX, EDX, and ECX are to be free for use within a procedure or function, and need not be preserved.

x86-64 calling conventions

x86-64 calling conventions take advantage of the additional register space to pass more arguments in registers. Also, the number of incompatible calling conventions has been reduced. There are two in common use.

Microsoft x64 calling convention

The Microsoft x64 calling convention^[14][15] is followed on Windows and pre-boot UEFI (for long mode on x86-64). It uses registers RCX, RDX, R8, R9 for the first four integer or pointer arguments (in that order), and XMM0, XMM1, XMM2, XMM3 are used for floating point arguments. Additional arguments are pushed onto the stack (right to left). Integer return values (similar to x86) are returned in RAX if 64 bits or less. Floating point return values are returned in XMM0. Parameters less than 64 bits long are not zero extended; the high bits are not zeroed.

When compiling for the x64 architecture in a Windows context (whether using Microsoft or non-Microsoft tools), there is only one calling convention – the one described here, so that stdcall, thiscall, cdecl, fastcall, etc., are now all one and the same.

In the Microsoft x64 calling convention, it is the caller's responsibility to allocate 32 bytes of "shadow space" on the stack right before calling the function (regardless of the actual number of parameters used), and to pop the stack after the call. The shadow space is used to spill RCX, RDX, R8, and R9,^[16] but must be made available to all functions, even those with fewer than four parameters.

The registers RAX, RCX, RDX, R8, R9, R10, R11 are considered volatile (caller-saved).^[17]

The registers RBX, RBP, RDI, RSI, RSP, R12, R13, R14, and R15 are considered nonvolatile (callee-saved).^[17]

For example, a function taking 5 integer arguments will take the first to fourth in registers, and the fifth will be pushed on the top of the shadow space. So when the called function is entered, the stack will be composed of (in ascending order) the return address, followed by the shadow space (32 bytes) followed by the fifth parameter.

In x86-64, Visual Studio 2008 stores floating point numbers in XMM6 and XMM7 (as well as XMM8 through XMM15); consequently, for x86-64, user-written assembly language routines must preserve XMM6 and XMM7 (as compared to x86 wherein user-written assembly language routines did not need to preserve XMM6 and XMM7). In other words, user-written assembly language routines must be updated to save/restore XMM6 and XMM7 before/after the function when being ported from x86 to x86-64.

Starting with Visual Studio 2013, Microsoft introduced the __vectorcall calling convention which extends the x64 convention.

System V AMD64 ABI

The calling convention of the System V AMD64 ABI is followed on Solaris, Linux, FreeBSD, macOS,^[18] and is the de facto standard among Unix and Unix-like operating systems. The first six integer or pointer arguments are passed in registers RDI, RSI, RDX, RCX (R10 in the Linux kernel interface^[19]:124), R8, and R9, while XMM0, XMM1, XMM2, XMM3, XMM4, XMM5, XMM6 and XMM7 are used for certain floating point arguments.^[19]:20 As in the Microsoft x64 calling convention, additional arguments are passed on the stack^[19]:20 and the return value is stored in RAX and RDX.^[19]:22

If the callee wishes to use registers RBP, RBX, and R12–R15, it must restore their original values before returning control to the caller. All other registers must be saved by the caller if it wishes to preserve their values.^[19]:15

If the callee is a variadic function, then the number of floating point arguments passed to the function in vector registers must be provided by the caller in the RAX register.^[19]:50

Unlike the Microsoft calling convention, a shadow space is not provided; on function entry, the return address is adjacent to the seventh integer argument on the stack.

List of x86 calling conventions

This is a list of x86 calling conventions.^[1] These are conventions primarily intended for C/C++ compilers (especially the 64-bit part below), and thus largely special cases. Other languages may use other formats and conventions in their implementations.

References

Footnotes

Other sources

Further reading