spiegel-keyman/docs/file-formats/kmx-file-format.md

KMX Binary Format

This document describes the binary format for KMX, compiled from Keyman .kmn source files.

Principles

• All integer values are unsigned 32-bit little-endian unless otherwise specified.
• All offsets are 32-bit little-endian values. All offsets are relative to the beginning of the data (normally start of file). A zero offset indicates that the value is not present, equivalent to NULL -- no offset can ever point to the start of the file.
• All strings are UTF-16LE unless otherwise specified and null terminated. Strings are referenced by offset (see above).
• The file format has a header section followed by data. There is no specific defined order for data in the file. Empty space is permitted but should be zeroed. Note that COMP_KEYBOARD_KMXPLUS must follow immediately after COMP_KEYBOARD with no empty space (that is at byte offset 64).

A .kmx file is essentially a 32-bit memory mapped file, however there is no guarantee that all sections start at a 32-bit boundary. In a 32-bit process on architectures that use flexible alignment, the file can be loaded into a memory buffer, and (assuming the file is valid), all offset values in the file can be rewritten into pointers by adding the base of the memory buffer to them. (In a 64-bit process or on architectures that use strict alignment (like emscripten), a fixup is needed to make space for aligned pointers.)

COMP_KEYBOARD: KMX header

The very first section is a header that describes the remainder of the file.

∆	Bits	Name	Description
0	32	dwIdentifier	KXTS 0x5354584B, kmx.FILEID_COMPILED
4	32	dwFileVersion	minimum Keyman version that can read the file, KMX_Version enum
8	32	dwCheckSum	deprecated in 16.0, 0x0; CRC32 checksum of entire file
12	32	KeyboardID	deprecated in 10.0, 0x0; Windows LANGID for keyboard
16	32	IsRegistered	deprecated in 10.0, 0x1; registration status of compiler
20	32	version	deprecated, 0x0; was version of keyboard, see TSS_KEYBOARDVERSION
24	32	cxStoreArray	number of COMP_STORE entries in COMP_KEYBOARD.dpStoreArray
28	32	cxGroupArray	number of COMP_GROUP entries in COMP_KEYBOARD.dpGroupArray
32	32	dpStoreArray	offset of first COMP_STORE entry
36	32	dpGroupArray	offset of first COMP_GROUP entry
40	32	StartGroup_ANSI	deprecated; index of starting non-Unicode COMP_GROUP, 0xFFFFFFFF means unused
44	32	StartGroup_Unicode	index of starting Unicode COMP_GROUP, 0xFFFFFFFF means unused
48	32	dwFlags	global flags for the keyboard, see description below
52	32	dwHotKey	default hotkey for keyboard, from &Hotkey, see description below
56	32	dpBitmapOffset	offset of keyboard icon
60	32	dwBitmapSize	size in bytes of keyboard icon, 0x0 if not present

This structure is present at the start of every .kmx file. The KXTS identifier can be used as a 'magic' to determine whether a binary file is a .kmx file.

Note that dpBitmapOffset should be ignored if dwBitmapSize is 0x0 -- it may be a non-zero value.

dwFileVersion: Minimum version of Keyman required to read the file

The dwFileVersion property should be checked before attempting to read any other data in the file. If it is not a recognized version, then no further attempt should be made to read the file. The recognized versions are listed in the KMX_Version enum:

Value	Identifier	Description
0x00000300	VERSION_30	Keyman 3.0. Non-Unicode, Windows 3.1 only, not supported
0x00000301	VERSION_31	Keyman 3.1. Non-Unicode, Windows 3.1 only, not supported
0x00000302	VERSION_32	Keyman 3.2. Non-Unicode, Windows 3.1, 95, 98 only, not supported
0x00000400	VERSION_40	Keyman 4.0. Non-Unicode, Windows 3.1, 95, 98 only, not supported
0x00000500	VERSION_50	Keyman 5.0. Unicode, Windows only
0x00000501	VERSION_501	Keyman 5.1, added support for call() statement
0x00000600	VERSION_60	Keyman 6.0
0x00000700	VERSION_70	Keyman 7.0
0x00000800	VERSION_80	Keyman 8.0
0x00000900	VERSION_90	Keyman 9.0
0x00000A00	VERSION_100	Keyman 10.0, first true multi-platform version
0x00000E00	VERSION_140	Keyman 14.0 (versions 11.0-13.0 had no binary format changes)
0x00000F00	VERSION_150	Keyman 15.0
0x00001000	VERSION_160	Keyman 16.0
0x00001100	VERSION_170	Keyman 17.0

StartGroup

Keyboards may have up to 4 starting groups, defined by begin statements in the source .kmn file.. The begin ANSI group and begin Unicode groups are defined in the COMP_KEYBOARD header, however as they were introduced into the format later, the begin NewContext and begin PostKeystroke groups are defined as system stores.

dwFlags

The following flags are defined for .kmx keyboards:

Bit value	Name	Description
0x00000001	KF_SHIFTFREESCAPS	Pressing Shift releases Caps Lock (see &ShiftFreesCaps)
0x00000002	KF_CAPSONONLY	Caps Lock switches on only (see &CapsOnOnly)
0x00000004	KF_CAPSALWAYSOFF	Caps Lock state is disabled (see &CapsAlwaysOff)
0x00000008	KF_LOGICALLAYOUT	Unused, should never be set
0x00000010	KF_AUTOMATICVERSION	The compiler determined the minimum version of Keyman automatically (see &Version)
0x00000020	KF_KMXPLUS	16.0+: A COMP_KEYBOARD_KMXPLUSINFO structure is present immediately after COMP_KEYBOARD

dwHotKey: keyboard hotkeys

This field is 0x0 if no default hotkey is specified. The value is a bitmask:

Bit mask	Name	Description
0x000000FF	virtual key	Keyman virtual key value of the hotkey
0x00010000	HK_ALT	Set if the hotkey uses Alt
0x00020000	HK_CTRL	Set if the hotkey uses Ctrl
0x00040000	HK_SHIFT	Set if the hotkey uses Shift

Keyboard icon

The COMP_KEYBOARD.dpBitmapOffset and COMP_KEYBOARD.dwBitmapSize members describe the keyboard icon, if present. If not present, both members should have the value 0x0. The keyboard icon may be a 16x16 Windows .bmp format file, 4, 8, 24, or 32 bit color, or a Windows .ico format file, which may have multiple icons present at different bit depths and resolutions. Refer to the Windows API documentation for .bmp and .ico binary formats.

By convention, the compiler stores the bitmap image at the end of the KMX data, but this is not a requirement of the file format.

COMP_KEYBOARD_KMXPLUSINFO: KMX+ secondary header

∆	Bits	Name	Description
0	32	dpKMXPlus	offset of KMX+ data
4	32	dwKMXPlusSize	size in bytes of KMX+ data

The COMP_KEYBOARD_KMXPLUSINFO structure is present only if COMP_KEYBOARD.dwFlags flag has bit KF_KMXPLUS (0x20) set. If present, it must be located immediately after the COMP_KEYBOARD structure, that is, at byte offset 64.

See KMX+ File Format for details of the KMX+ file format.

COMP_STORE

∆	Bits	Name	Description
0	32	dwSystemID	0x0 for a normal store, or a system store value
4	32	dpName	offset to name of the store from keyboard source, 0x0 if not present
8	32	dpString	offset to value of the store

The COMP_STORE structure describes a store in the keyboard source file. COMP_STORE structures are also used for additional metadata such as compiler version.

Store names are compiled into keyboards if debug information is included (see Debug information for keyboards), or if the store is used for keyboard options. If the store name is not included, dpName will have a 0x0 value.

System Stores

If dwSystemID is non-zero, then it will be one of the following values:

Value	Identifier	Count	Version	Description
0	TSS_NONE	0+	5.0	(Not a system store, normal keyboard store data)
1	TSS_BITMAP	0-1	5.0	&Bitmap store, used during compile only; see COMP_KEYBOARD.dpBitmapOffset
2	TSS_COPYRIGHT	0-1	5.0	&Copyright store
3	TSS_HOTKEY	0-1	5.0	&Hotkey store (see also COMP_KEYBOARD.dwHotkey)
4	TSS_LANGUAGE	0-1	5.0	Deprecated, &Language store
5	TSS_LAYOUT	0-1	5.0	Deprecated, LAYOUT statement
6	TSS_MESSAGE	0-1	5.0	&Message store
7	TSS_NAME	0-1	5.0	&Name store, public name of keyboard
8	TSS_VERSION	0-1	5.0	&Version store; use COMP_KEYBOARD.dwFileVersion
9	TSS_CAPSONONLY	0-1	5.0	&CapsOnOnly store; use TF_CAPSONONLY
10	TSS_CAPSALWAYSOFF	0-1	5.0	&CapsAlwaysOff store; use TF_CAPSALWAYSOFF
11	TSS_SHIFTFREESCAPS	0-1	5.0	&ShiftFreesCaps store; use TF_SHIFTFREESCAPS
12	TSS_LANGUAGENAME	0-1	5.0	Deprecated, LANGUAGENAME statement
13	TSS_CALLDEFINITION	0+	5.0	Definition for an IMX call
14	TSS_CALLDEFINITION_LOADFAILED	0	5.0	Deprecated, unused, was used internally in KMX processor
15	TSS_ETHNOLOGUECODE	0-1	5.0	Deprecated, &EthnologueCode store
16	TSS_DEBUG_LINE	0+	5.0	Debug information for keyboards
17	TSS_MNEMONIC	0-1	5.0	&MnemonicLayout store, '0' or '1'
18	TSS_INCLUDECODES	0-1	5.0	&IncludeCodes store, used during compile only
19	TSS_OLDCHARPOSMATCHING	0-1	5.0	Deprecated, &OldCharPosMatching store
20	TSS_COMPILEDVERSION	0-1	5.0	Version of compiler used, may be omitted if --no-compiler-version passed to kmc
21	TSS_KEYMANCOPYRIGHT	0-1	5.0	Copyright message from compiler, may be omitted if --no-compiler-version passed to kmc
22	TSS_CUSTOMKEYMANEDITION	1	5.0	Deprecated, always '0', always present
23	TSS_CUSTOMKEYMANEDITIONNAME	1	5.0	Deprecated, always 'Keyman', always present
24	TSS_VISUALKEYBOARD	0-1	7.0	&VisualKeyboard store
25	TSS_KMW_RTL	0-1	7.0	&KMW_RTL store, '0' or '1'
26	TSS_KMW_HELPFILE	0-1	7.0	&KMW_HelpFile store, used during compile only
27	TSS_KMW_HELPTEXT	0-1	7.0	&KMW_HelpText store, used during compile only
28	TSS_KMW_EMBEDJS	0-1	7.0	&KMW_EmbedJS store, used during compile only
29	TSS_WINDOWSLANGUAGES	0-1	7.0	Deprecated, &WindowsLanguages store
30	TSS_COMPARISON	0+	8.0	Generated by compiler for use in if() statements
31	TSS_PLATFORM	0-1	9.0	Cannot be used in COMP_STORE; used for if(&platform). platform() only
32	TSS_BASELAYOUT	0-1	9.0	Cannot be used in COMP_STORE; used for if(&baselayout), baselayout() only
33	TSS_LAYER	0	9.0	Cannot be used in COMP_STORE; used for if(&layer) and set(&layer)
34	TSS_VKDICTIONARY	0-1	9.0	Dictionary of T_ and U_ virtual key names for touch layouts, generated by compiler
35	TSS_LAYOUTFILE	0-1	9.0	Name of touch layout file, used during compile only
36	TSS_KEYBOARDVERSION	0-1	9.0	&KeyboardVersion store
37	TSS_KMW_EMBEDCSS	0-1	9.0	&KMW_EmbedCSS store, used during compile only
38	TSS_TARGETS	0-1	9.0	&Targets store
39	TSS_CASEDKEYS	0-1	14.0	&CasedKeys store, used during compile only
40	TSS_BEGIN_NEWCONTEXT	0-1	15.0	zero-based index of group referenced in begin NewContext
41	TSS_BEGIN_POSTKEYSTROKE	0-1	15.0	zero-based index of group referenced in begin PostKeystroke
42	TSS_NEWLAYER	0	15.0	Cannot be used in COMP_STORE; used for testing layer switch events
43	TSS_OLDLAYER	0	15.0	Cannot be used in COMP_STORE; used for testing layer switch events
44	TSS_DISPLAYMAP	0-1	17.0	&DisplayMap store

COMP_GROUP

∆	Bits	Name	Description
0	32	dpName	offset to name of the group from keyboard source, 0x0 if not present
4	32	dpKeyArray	offset to array of COMP_KEY entries
8	32	dpMatch	offset to string for the match statement output, 0x0 if not present
12	32	dpNoMatch	offset to string for the nomatch statement output, 0x0 if not present
16	32	cxKeyArray	number of COMP_KEY entries in COMP_GROUP.dpKeyArray
20	32	fUsingKeys	0x1 if group is using keys, 0x0 otherwise

The COMP_GROUP structure describes a group in the keyboard source file. Note that the readonly property of a group is only applicable in the source file and is never written to the binary format.

COMP_KEY

∆	Bits	Name	Description
0	16	Key	Keyman virtual key, virtual character key, or character, for the rule, 0 if unused
2	16	(unused)	padding, reserved
4	32	Line	Line number for the rule, 0 if not compiled with debug information
8	32	ShiftFlags	Modifier flags for the key, see description below
12	32	dpOutput	offset to string for the output of the rule, always present even for zero-length output
16	32	dpContext	offset to string for the context part of the rule, always present even for zero-length context

The COMP_KEY structure describes a single rule in a keyboard source file. The compiler expands any() statements found in the key part of a rule into multiple rules -- so the key part of the rule is always only ever matching a single key combination.

ShiftFlags contains a set of modifier and state key flags, and some additional flags that determine how to interpret Key.

Modifier and state key flags

The modifier and state key flags are:

Bit mask	Name	Description
0x00000001	LCTRLFLAG	Left Control key
0x00000002	RCTRLFLAG	Right Control key
0x00000004	LALTFLAG	Left Alt key (Option on macOS)
0x00000008	RALTFLAG	Right Alt key
0x00000010	K_SHIFTFLAG	Either Shift key
0x00000020	K_CTRLFLAG	Either Ctrl key
0x00000040	K_ALTFLAG	Either Alt key
0x00000080	K_METAFLAG	Either Meta key. Reserved, not for use in .kmx
0x00000100	CAPITALFLAG	Caps lock on
0x00000200	NOTCAPITALFLAG	Caps lock NOT on
0x00000400	NUMLOCKFLAG	Num lock on
0x00000800	NOTNUMLOCKFLAG	Num lock NOT on
0x00001000	SCROLLFLAG	Scroll lock on
0x00002000	NOTSCROLLFLAG	Scroll lock NOT on

• The chiral (left/right) modifier flags should not be used together with the non-chiral flags.
• If neither CAPITALFLAG nor NOTCAPITALFLAG are set, then the state of the Caps Lock key is ignored, and the same principle applies for Num lock and Scroll lock.
• The Meta key may be the Windows key on Windows, or the Command key on macOS. This flag is reserved for internal use, and is not valid in a .kmx file.

Additional flags

The meaning of Key is determined by the flags ISVIRTUALKEY and VIRTUALCHARKEY in ShiftFlags (mask 0x0000C000):

• 0: Key is a character on a key cap (US English unless the keyboard is a mnemonic layout). All other shift flags are ignord
• ISVIRTUALKEY: Key is a US English virtual key
• VIRTUALCHARKEY: Key is a character on a key cap, combined with the shift flags (US English unless the keyboard is a mnemonic layout)

Bit mask	Name	Description
0x00004000	ISVIRTUALKEY	Key is a virtual key
0x00008000	VIRTUALCHARKEY	Keyman 6.0+: Key is a Virtual character key

Note that the shift flags for key rules do not use the same values as hotkeys for historical reasons.

Debug information for keyboards

When debug information is present, it will be found in four places in compiled keyboards:

• COMP_STORE.dpName: the name of the store from the .kmn file
• COMP_GROUP.dpName: the name of the group from the .kmn file
• COMP_KEY.Line: the line number for the rule
• System stores with COMP_STORE.dwSystemID value of TSS_DEBUG_LINE

When a keyboard is compiled with debug data, the TSS_DEBUG_LINE system store contains debug information about each source line in the file. In each entry, COMP_STORE.dpName contains debug metadata, and COMP_STORE.dpString contains only the stringified .kmn source line number where the metadata is found.

The name of the TSS_DEBUG_LINE system store starts with a single character which describes the rest of the debug metadata in that store:

Character	Debug data type	Metadata
'B'	begin statement	Unicode or ANSI
'M'	match statement	Stringified index of containing group, space, group name
'N'	nomatch statement	Stringified index of containing group, space, group name
'G'	group statement	Stringified index of group, space, group name
'D'	deadkey statement	Stringified index of deadkey, space, deadkey name