calling convention

x68 calling conventions

stdcall

Function arguments are passed (pushed) from right to left. Stack is cleaned up by callee.

Return value is passed through the eax register (x86 assembler).

Stdcall is the default calling convention for the 32-bit WinAPI (after pascal was abandoned) and should apparently also be used for exported functions in DLLs.

Stdcall is almost the same as cdecl with the exception that the callee resets ESP to the initial state with ret x (instead of ret?). x = number of arguments * sizeof(int).

push arg3
push arg2
push arg1
call f

f:
  ; … do this , do that …
ret 12

cdecl

The default (native) calling convention for C and C++ programs. cdecl is short for c declaration.

Caller pushes arguments on stack and cleans up stack.

Cleaning up the stack causes larger programs than those with stdcall.

cdecl is required for variadic function arguments (vararg).

push arg3  ; rightmost argument
push arg2
push arg1  ; leftmost argument
call f
add esp, 12  ; 12 = 3 arguments each being 4 bytes

When entering the function (just after call f and before add esp, 12), the stack is:

		Notation in IDA
ESP	return address
ESP+4	arg1	`arg_0`
ESP+8	arg2	`arg_1`
ESP+12	arg3	`arg_2`

fastcall

Some values passed via registers

The 64-bit WinAPI is similar to the 32-bit WinAPI fastcall. The first four parameters are passed in rcx, rdx, r8 and r9 (from right to left). Additional parameters are stored on the stack.

thiscall

The (C++) this pointer is stored in the ecx register, other parameters are pushed on the stack.

pascal

The pascal calling convention differs from stdcall in that the function arguments are pushed left to right.

Name decoration

Both, MSVC and MinGW decorate stdcall and cdecl function names by prefixing them with an underscore. Additionally, stdcall function names will be followed by an at-sign and the number of bytes of the function parameters.

Apparently, with the linker option --kill-at, the suffixes are not created for stdcall DLL functions.

With --add-stdcall-alias, the function is exported with and without this suffix.

x64

Windows

Unlike x86, on a x64, there is only one calling convention. It takes advantage of the increased number of registers that are available.

The rules are

The first four integer or pointer parameters are passed in the rcx, rdx, r8 and r9 registers.
The first four floating-point parameters are passed in the first four SSE registers, xmm0 - xmm3
The caller reserves space on the stack for arguments passed in registers. The called function can use this space to spill the contents of registers to the stack
Any additional arguments are passed on the stack
An integer or pointer return value is returned in the rax register; a floating-point return value is returned in xmm0.
rax, rcx, rdx, r8 through r11, xmm0 through xmm15 are volatile
rbx, rbp, rdi, rsi, rsp and r12 through r15 are nonvolatile: these must be saved and restored by a function that uses them.

In C++, the this pointer is implicitly passed as first argument

Linux (System V ABI)

Arguments are passed similarly to Windows x64, but using 6 registers instead of 4: rdi, rsi, rdx, rcx, r8 and r9.

Preserved register (or callee-saved registers): The registers that must have the same value when a function returns as they had when the function was called are: rbx, rsp, r12, r13, r14, r15.

Returning floats and doubles

All calling conventions except Win64 return a float or a double in the fpu register st(0).

Win64 returns them in the low 32 or full 64 bits of the xmm0 register.