Search notes:

Windows subsystems

A subsystem exposes a subset of the base Windows executive system services to applications.
There are a few possible subsystems:
When linking, the subsystem is indicated with the /SUBSYSTEM option.

Console vs GUI

The arguably two most important subsystems are Windows and Console.
Because console applications are fundamentally different from GUI applications, there are some programs that come in the console and the gui versions: Java has java.exe and javaw.exe, the script host comes with cscript.exe and wscript and devenv has devenv.com and devenv.exe etc.
When using gcc (mingw), a console subsystem is created by specifing the -mconsole option.

Determining the substystem in a running Exe or DDL

The vim sources contain the function mch_is_gui_executable() in src/os_win32.c that determines the subsystem of the running vim executable/DLL:
    int
mch_is_gui_executable(void)
{
    PBYTE		pImage = (PBYTE)GetModuleHandle(NULL);
    PIMAGE_DOS_HEADER	pDOS = (PIMAGE_DOS_HEADER)pImage;
    PIMAGE_NT_HEADERS	pPE;

    if (pDOS->e_magic != IMAGE_DOS_SIGNATURE)
	return FALSE;
    pPE = (PIMAGE_NT_HEADERS)(pImage + pDOS->e_lfanew);
    if (pPE->Signature != IMAGE_NT_SIGNATURE)
	return FALSE;
    if (pPE->OptionalHeader.Subsystem == IMAGE_SUBSYSTEM_WINDOWS_GUI)
	return TRUE;
    return FALSE;
}

See also

Windows 10, version 1607 comes with a Windows Subsystem for Linux (developers only). This is not a »real« subsystem like Console or GUI.
Subsystems are started by smss.exe (Windows Session Manager)
See also value Required under registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\SubSystems.

Index