Search notes:

calling convention

C calling convention @ calling
The calling convention states how arguments (parameters) are passed to a function and how the caller receives a return value. Amont others, the convention regulates if those values are passed via CPU registers and/or via the stack and which registers are guaranteed to preserve their values across function calls.

x68 calling conventions

stdcall

Function arguments are passed (pushed) from right to left. Stack is cleaned up by callee.
Return value is passed through the eax register (x86 assembler).
Stdcall is the default calling convention for the 32-bit WinAPI (after pascal was abandoned) and should apparently also be used for exported functions in DLLs.
Stdcall is almost the same as cdecl with the exception that the callee resets ESP to the initial state with ret x (instead of ret?). x = number of arguments * sizeof(int). 
push arg3
push arg2
push arg1
call f

f:
  ; … do this , do that …
ret 12

cdecl

The default (native) calling convention for C and C++ programs. cdecl is short for c declaration.
Caller pushes arguments on stack and cleans up stack.
Cleaning up the stack causes larger programs than those with stdcall.
cdecl is required for variadic function arguments (vararg). 
push arg3  ; rightmost argument
push arg2
push arg1  ; leftmost argument
call f
add esp, 12  ; 12 = 3 arguments each being 4 bytes
When entering the function (just after call f and before add esp, 12), the stack is:
Notation in IDA
ESP return address
ESP+4 arg1 arg_0
ESP+8 arg2 arg_1
ESP+12 arg3 arg_2

fastcall

Some values passed via registers
The 64-bit WinAPI is similar to the 32-bit WinAPI fastcall. The first four parameters are passed in rcx, rdx, r8 and r9 (from right to left). Additional parameters are stored on the stack.

thiscall

The (C++) this pointer is stored in the ecx register, other parameters are pushed on the stack.

pascal

The pascal calling convention differs from stdcall in that the function arguments are pushed left to right.

Name decoration

Both, MSVC and MinGW decorate stdcall and cdecl function names by prefixing them with an underscore. Additionally, stdcall function names will be followed by an at-sign and the number of bytes of the function parameters.
Apparently, with the linker option --kill-at, the suffixes are not created for stdcall DLL functions.
With --add-stdcall-alias, the function is exported with and without this suffix.

x64

Windows

Unlike x86, on a x64, there is only one calling convention. It takes advantage of the increased number of registers that are available.
The rules are
  • The first four integer or pointer parameters are passed in the rcx, rdx, r8 and r9 registers.
  • The first four floating-point parameters are passed in the first four SSE registers, xmm0 - xmm3
  • The caller reserves space on the stack for arguments passed in registers. The called function can use this space to spill the contents of registers to the stack
  • Any additional arguments are passed on the stack
  • An integer or pointer return value is returned in the rax register; a floating-point return value is returned in xmm0.
  • rax, rcx, rdx, r8 through r11, xmm0 through xmm15 are volatile
  • rbx, rbp, rdi, rsi, rsp and r12 through r15 are nonvolatile: these must be saved and restored by a function that uses them.
In C++, the this pointer is implicitly passed as first argument

Linux

Arguments are passed similarly to Windows x64, but using 6 registers instead of 4: rdi, rsi, rdx, rcx, r8 and r9.

Returning floats and doubles

All calling conventions except Win64 retourn a float or a double in the fpu register st(0).
Win64 returns them in the low 32 or full 64 bits of the xmm0 register.

See also

The System.Runtime.InteropServices.CallingConvention enum.

Index