Bitstream Format

Kanzi Bitstream Specification (Version 6)

This document defines the bitstream format for the Kanzi lossless data compressor. A Kanzi bitstream can be either standard (containing a global header) or headless.

1. Introduction

This document specifies the Kanzi bitstream format, which defines how compressed data produced by the Kanzi lossless data compressor is serialized into a binary representation suitable for storage or transmission.

A bitstream is an ordered sequence of bits that is interpreted sequentially from beginning to end. In Kanzi, the bitstream represents a complete compressed data stream and is decoded strictly in forward order.

A Kanzi bitstream is composed of the following elements, in order:

• An optional global header, present only in standard mode

• A sequence of one or more compressed blocks

• A mandatory End of Stream marker

In standard mode, the global header appears once at the beginning of the bitstream and fully specifies all parameters required to decode the subsequent blocks, including the entropy codec, the list of transforms, the block size, and the checksum type.

In headless mode, the global header is omitted. In this case, the decoder must be configured externally with the same parameters that would otherwise be provided by the header. The structure and interpretation of the compressed blocks are identical in both modes.

Each block is compressed independently and can be decoded without reference to other blocks, except for the shared configuration defined by the global header or the external parameters in headless mode. This block-based design enables streaming operation, parallel decoding, and partial recovery in the presence of data corruption.

Unless explicitly stated otherwise, all multi-byte numeric fields in the bitstream are stored in Big-Endian order, and all numeric values are unsigned.

2. Global Header

Except in headless mode, the global header is present once at the beginning of the stream.

All multi-byte values are stored in Big-Endian order.

Bits	Name	Value	Description
32	Magic Number	0x4B414E5A	'KANZ' constant
4	bsVersion	6	Bitstream format version
2	chkSize	0..2	Block checksum: 0=None, 1=32-bit, 2=64-bit, 3=Reserved
5	entropyType	0..31	Entropy codec identifier (see Section 5.1)
48	transformType	-	Eight 6-bit transform identifiers (see Section 5.2)
28	blockSize	-	Block size divided by 16 (1024 to 1GB)
2	szMask	0..3	Size of the optional output size field
0,16,32,48	outputSize	-	Original uncompressed size (present only if szMask != 0)
15	Padding	0	Reserved for future use (must be 0)
24	Checksum	-	Header verification checksum

2.1 Output Size Encoding

The _outputSize field is present only if szMask is non-zero:

szMask = 0: no output size field

szMask = 1: 16-bit output size

szMask = 2: 32-bit output size

szMask = 3: 48-bit output size

The output size represents the total uncompressed size in bytes.

2.2 Header Checksum Calculation

The 24-bit header checksum is calculated as follows:

crcSize := uint(24)
seed := uint32(0x01030507 * bsVersion)
var cksum uint32
HASH := uint32(0x1E35A7BD)
cksum = HASH * seed
cksum ^= (HASH * uint32(^ckSize))
cksum ^= (HASH * uint32(^this.entropyType))
cksum ^= (HASH * uint32((^this.transformType)>>32))
cksum ^= (HASH * uint32(^this.transformType))
cksum ^= (HASH * uint32(^this.blockSize))

if szMask > 0 {
   cksum ^= (HASH * uint32((^this.outputSize)>>32))
   cksum ^= (HASH * uint32(^this.outputSize))
}

cksum = (cksum >> 23) ^ (cksum >> 3)
cksum &= ((1 << crcSize) - 1))

3. Block Format

After the global header, one or more blocks follow.

Each block is independently encoded and decoded.

3.1 Block Header

Bits	Name	Description
5	logSize	Value L = log2(compressed_block_size) - 3
L + 3	cbs	Compressed block size in bits. If 0, this signals End of Stream

3.2 Block Data

Bits	Name	Description
8	Mode	Block flags and pre-transform size descriptor (see 3.3)
0 or 8	SkipFlags	Present only if bit 4 of Mode is 1
8 * ps	DataSize	Encoded size of data before transforms. ps = 1 + ((Mode >> 5) & 0x03) bytes
0,32,64	BlockChecksum	XXHash32 or XXHash64 of decompressed data (seed = 0x4B414E5A)

3.3 Mode Byte and Skip Flags

The Mode byte is bit-packed as follows:

Bit 7: Copy block flag If set, the block data is stored verbatim (no entropy coding or transforms).

Bits 6-5: Pre-transform size field length minus 1 (values 0 to 3)

Bit 4: Skip flags location

1: Skip flags are read as the next 8 bits in the bitstream

0: Skip flags are derived from the lower nibble of the Mode byte

Bits 3-0: Inline skip flags for the first four transforms

A bit set to 1 indicates that the corresponding transform is skipped.

Skip Flags Resolution:

If Bit 4 is 0: FinalSkipFlags = (Mode << 4) | 0x0F (Transforms 5 to 8 are implicitly skipped)

If Bit 4 is 1: FinalSkipFlags is the explicitly read 8-bit SkipFlags value

4. Decompression Process

For each block:

Read the block header and extract the compressed block size.

If cbs == 0, stop decoding (End of Stream).

Read the Mode byte and determine the FinalSkipFlags.

If Bit 7 of Mode is set (Copy Block):

Copy the block payload directly to the output.

Otherwise (Compressed Block):

Entropy-decode the data using the codec specified in the header.

Apply inverse transforms sequentially from last to first, skipping transforms whose corresponding bit in FinalSkipFlags is set.

If chkSize > 0 (a block checksum is available), compute the XXHash checksum of the decompressed block and compare it with the BlockChecksum stored in the bitstream.

5. Constants

5.1 Entropy Types

ID	Name
0	NONE
1	HUFFMAN
2	FPAQ
3	PAQ (obsolete)
4	RANGE
5	ANS0
6	CM
7	TPAQ
8	ANS1
9	TPAQX

5.2 Transform Types

ID	Name
0	NONE
1	BWT
2	BWTS
3	LZ
4	Snappy (obsolete)
5	RLT
6	ZRLT
7	MTFT
8	RANK
9	EXE
10	DICT
11	ROLZ
12	ROLZX
13	SRT
14	LZP
15	MM
16	LZX
17	UTF
18	PACK
19	DNA

6. End of Stream

The bitstream must terminate with an empty block.

This is encoded as a single null byte (0x00), representing 5 zero bits for logSize and 3 zero bits for cbs.

Kanzi Bitstream Specification (Version 7)

Version 7 preserves the global header layout, block framing, block checksum, and End of Stream marker of version 6. It changes the header checksum mixing function, adds a checksum to each block header, and introduces transformed-copy blocks.

A Kanzi bitstream can be either standard (containing a global header) or headless. Unless explicitly stated otherwise, all multi-byte numeric fields are stored in Big-Endian order, and all numeric values are unsigned.

1. Introduction

A Kanzi bitstream is composed of the following elements, in order:

• An optional global header, present only in standard mode

• A sequence of one or more compressed blocks

• A mandatory End of Stream marker

In standard mode, the global header specifies the entropy codec, transform sequence, block size, checksum type, and optional original size. In headless mode, these parameters are supplied externally. Each block is independently encoded and decoded.

2. Global Header

Except in headless mode, the global header is present once at the beginning of the stream.

The field layout is unchanged from version 6:

Bits	Name	Value	Description
32	Magic Number	0x4B414E5A	'KANZ' constant
4	bsVersion	7	Bitstream format version
2	chkSize	0..2	Block checksum: 0=None, 1=32-bit, 2=64-bit, 3=Reserved
5	entropyType	0..31	Entropy codec identifier (see Section 5.1)
48	transformType	-	Eight 6-bit transform identifiers (see Section 5.2)
28	blockSize	-	Block size divided by 16 (1024 to 1GB)
2	szMask	0..3	Size of the optional output size field
0,16,32,48	outputSize	-	Original uncompressed size (present only if szMask != 0)
15	Padding	0	Reserved for future use (must be 0)
24	Checksum	-	Header verification checksum

2.1 Output Size Encoding

The _outputSize field is present only if szMask is non-zero:

szMask = 0: no output size field

szMask = 1: 16-bit output size

szMask = 2: 32-bit output size

szMask = 3: 48-bit output size

The output size represents the total uncompressed size in bytes.

2.2 Header Checksum Calculation

The 24-bit header checksum uses a chained 32-bit mixing function. The values passed to the mixing function are not complemented; the function performs the complement internally.

func mix32(cksum, hash, value uint32) uint32 {
   cksum ^= hash * ^value
   cksum = (cksum << 13) | (cksum >> 19)
   return cksum*5 + 0x52DCE729
}

seed := uint32(0x01030507 * bsVersion)
HASH := uint32(0x1E35A7BD)
cksum := HASH * seed
cksum = mix32(cksum, HASH, uint32(chkSize))
cksum = mix32(cksum, HASH, uint32(entropyType))
cksum = mix32(cksum, HASH, uint32(transformType >> 32))
cksum = mix32(cksum, HASH, uint32(transformType))
cksum = mix32(cksum, HASH, uint32(blockSize))

if szMask > 0 {
   cksum = mix32(cksum, HASH, uint32(outputSize >> 32))
   cksum = mix32(cksum, HASH, uint32(outputSize))
}

cksum = (cksum >> 23) ^ (cksum >> 3)
cksum &= ((1 << 24) - 1)

3. Block Format

After the global header, one or more blocks follow. Each block is independently encoded and decoded.

3.1 Block Header

Bits	Name	Description
5	logSize	Value L = log2(compressed_block_size) - 3
L + 3	cbs	Compressed block size in bits. If 0, this signals End of Stream

The cbs value is also covered by the block header checksum described in Section 3.4.

3.2 Block Data

Bits	Name	Description
8	Mode	Block flags and pre-transform size descriptor (see 3.3)
0 or 8	SkipFlags	Explicit transform skip flags when required (see 3.3)
8 * ps	DataSize	Size of data before inverse transforms. ps = 1 + ((Mode >> 5) & 0x03) bytes
8	HeaderChecksum	Checksum of the block header and cbs (see 3.4)
0,32,64	BlockChecksum	XXHash32 or XXHash64 of decompressed data (seed = 0x4B414E5A)
variable	Payload	Raw transformed data or entropy-coded data

3.3 Mode Byte and Skip Flags

The Mode byte is bit-packed as follows:

Bit 7: Copy block flag

If set and Bit 4 is clear, the payload is a raw copy: it contains untransformed, unentropy-coded data.

If set and Bit 4 is set, the block is a transformed-copy block. The payload contains the output of the configured transform sequence, stored without entropy coding. The decoder reads the payload and applies the inverse transforms.

Bits 6-5: Pre-transform size field length minus 1 (values 0 to 3)

Bit 4: Skip flags location or transformed-copy indicator

For an entropy-coded block (Bit 7 clear):

1: Skip flags are read as the next 8 bits in the bitstream

0: Skip flags are derived from the lower nibble of the Mode byte

For a transformed-copy block (Bits 7 and 4 set):

If the transform sequence contains more than four transforms, skip flags are read as the next 8 bits in the bitstream

Otherwise, skip flags are derived from the lower nibble of the Mode byte

Bits 3-0: Inline skip flags for the first four transforms

A bit set to 1 indicates that the corresponding transform is skipped.

Skip Flags Resolution:

If no explicit skip-flags byte is present, FinalSkipFlags = (Mode << 4) | 0x0F. If an explicit byte is present, FinalSkipFlags is that 8-bit value. For a raw copy block, transforms are not applied.

3.4 Block Header Checksum

The 8-bit HeaderChecksum is calculated after the complete block has been encoded because it includes cbs, the compressed block length in bits. The checksum covers the Mode byte, the resolved FinalSkipFlags, the DataSize value, and cbs; it does not cover the optional BlockChecksum or the payload.

HASH := uint32(0x1E35A7BD)
seed := uint32(0x01030507)
cksum := HASH * seed
cksum = mix32(cksum, HASH, uint32(mode))
cksum = mix32(cksum, HASH, uint32(finalSkipFlags))
cksum = mix32(cksum, HASH, uint32(dataSize))
cksum = mix32(cksum, HASH, uint32(cbs >> 32))
cksum = mix32(cksum, HASH, uint32(cbs))
cksum = (cksum >> 23) ^ (cksum >> 3)
headerChecksum = cksum & 0xFF

The checksum is verified before the remaining block payload is read or decoded.

4. Decompression Process

For each block:

Read the block header and extract the compressed block size.

If cbs == 0, stop decoding (End of Stream).

Read the Mode byte, any explicit SkipFlags byte, and the DataSize field. Read and verify HeaderChecksum before processing the payload.

If Bit 7 of Mode is set and Bit 4 is clear (raw copy block):

Read the payload directly to the output. No entropy decoding or inverse transform is performed.

If Bits 7 and 4 of Mode are set (transformed-copy block):

Read the transformed payload directly.

Apply inverse transforms sequentially from last to first, skipping transforms whose corresponding bit in FinalSkipFlags is set.

Otherwise (entropy-coded block):

Entropy-decode the payload using the codec specified in the global header or headless parameters.

Apply inverse transforms sequentially from last to first, skipping transforms whose corresponding bit in FinalSkipFlags is set.

If chkSize > 0 (a block checksum is available), compute the XXHash checksum of the decompressed block and compare it with the BlockChecksum stored in the bitstream.

5. Constants

The entropy and transform identifiers are unchanged from version 6.

5.1 Entropy Types

ID	Name
0	NONE
1	HUFFMAN
2	FPAQ
3	PAQ (obsolete)
4	RANGE
5	ANS0
6	CM
7	TPAQ
8	ANS1
9	TPAQX

5.2 Transform Types

ID	Name
0	NONE
1	BWT
2	BWTS
3	LZ
4	Snappy (obsolete)
5	RLT
6	ZRLT
7	MTFT
8	RANK
9	EXE
10	DICT
11	ROLZ
12	ROLZX
13	SRT
14	LZP
15	MM
16	LZX
17	UTF
18	PACK
19	DNA

6. End of Stream

The bitstream must terminate with an empty block.

This is encoded as a single null byte (0x00), representing 5 zero bits for logSize and 3 zero bits for cbs.