synology_apm_repo.sdk.format.bucket module

Bucket file (.buk) format (FORMAT-SPEC.md §4).

Pure bytes -> dataclass decode, zero I/O. The Dedup Layer’s BucketReader fetches bytes at the offsets this module computes, then hands ciphertext through format.crypto and compression.

In practice mode is COMPRESS|CHUNK_CRC (0x03), plus VAULT_ENCRYPT (0x83) for an encrypted bucket, but readers test each bit rather than assume it (FORMAT-SPEC.md: Header & mode bits).

synology_apm_repo.sdk.format.bucket.COMPRESS_TYPE_BY_VALUE = {0: CompressType.NONE, 1: CompressType.LZ4, 2: CompressType.ZSTD, 4: CompressType.COMPACTED}

Plain-dict stand-in for CompressType(value) in the per-chunk hot loops (up to 8192 calls per bucket), avoiding Enum.__call__ overhead.

synology_apm_repo.sdk.format.bucket.COMPRESS_TYPE_COMPACTED_VALUE = 4

CompressType.COMPACTED’s raw value, for hot loops that compare raw SizeStore values (BucketIndex, BucketReader) without building CompressType members.

class synology_apm_repo.sdk.format.bucket.BucketFileHeader(major, minor, mode, chunk_num, chunk_size_crc, crc_of_chunk_crc)

Bases: object

Parsed .buk header (FORMAT-SPEC.md: Header & mode bits).

major: int
minor: int
mode: int
chunk_num: int
chunk_size_crc: int
crc_of_chunk_crc: int

CRC32 over the whole ChunkCrcStore trailer (FORMAT-SPEC.md: ChunkCrcStore & Redundancy); not needed to read a chunk, checked when the trailer is loaded (parse_chunk_crc_store’s verify_crc).

property is_compressed: bool
property is_vault_encrypted: bool

The only reliable signal that this bucket’s chunks are encrypted; repo_info’s encrypt_algorithm says nothing about an individual bucket (FORMAT-SPEC.md: Header & mode bits).

synology_apm_repo.sdk.format.bucket.parse_bucket_header(data)

Parse a .buk file’s 64-byte header from the start of data.

Raises:
class synology_apm_repo.sdk.format.bucket.SizeStoreEntry(compress_type, stored_len)

Bases: NamedTuple

One chunk’s compression type and stored (compressed) length, as recorded in SizeStore (FORMAT-SPEC.md: SizeStore).

A NamedTuple, not a frozen dataclass: built up to 8192 times per bucket, where it is cheaper.

compress_type: CompressType

Alias for field number 0

stored_len: int

The raw 12-bit size field. Not the same as effective_len.

property effective_len: int

Actual bytes this chunk occupies in the data region. 4096 for CompressType.NONE (stored_len is meaningless there), 0 for CompressType.COMPACTED, otherwise stored_len verbatim.

synology_apm_repo.sdk.format.bucket.chunk_size_store_tight_length(chunk_num)

ceil(chunk_num * 15 / 8), the tightly-packed SizeStore length in bytes, before zero-padding to the fixed 16320-byte on-disk allocation.

synology_apm_repo.sdk.format.bucket.parse_size_store(data, chunk_num, *, verify_crc=None)

Decode chunk_num 15-bit packed SizeStore records starting at the beginning of data (FORMAT-SPEC.md: SizeStore, the bucket’s SizeStore region, bytes [64, 16384)); only the first chunk_size_store_tight_length bytes are read.

Every record’s CompressType is validated eagerly; the result is the compressed layout’s BucketIndex.

Parameters:
  • data (bytes) – Bytes starting at the SizeStore region.

  • chunk_num (int) – Number of records to decode, from the bucket header.

  • verify_crc (int | None) – The header’s chunkSizeCrc field. When given, the tightly-packed bytes’ CRC32 is checked against it.

Raises:
  • FormatError – data is shorter than the tightly-packed region chunk_num implies.

  • DataCorruptError – verify_crc was given and does not match, or a decoded record’s CompressType value is unknown.

class synology_apm_repo.sdk.format.bucket.ChunkLocator(offset, length)

Bases: NamedTuple

Absolute file byte-range for one chunk’s (still compressed and/or encrypted) data. A NamedTuple, not a frozen dataclass: built per chunk read, where it is cheaper.

offset: int

Alias for field number 0

length: int

Alias for field number 1

class synology_apm_repo.sdk.format.bucket.BucketIndex(compress_types, stored_lens, effective_lens, offsets)

Bases: Sequence[SizeStoreEntry]

One bucket’s per-chunk SizeStore entries and data locations, as parallel array.array columns (FORMAT-SPEC.md: SizeStore, Physical layout). A Sequence[SizeStoreEntry]; locator(i) is chunk i’s byte range. Hot loops read the columns directly; they are shared, so never mutate them.

Variables:
  • compress_types (array.array[int]) – Raw CompressType values ("B").

  • stored_lens (array.array[int]) – Raw 12-bit size fields ("H").

  • effective_lens (array.array[int]) – Bytes each chunk occupies in the data region (SizeStoreEntry.effective_len; "H"); also each locator’s length.

  • offsets (array.array[int]) – Each chunk’s absolute file offset ("I": a bucket tops out at ~33.6 MB).

compress_types: array[int]
stored_lens: array[int]
effective_lens: array[int]
offsets: array[int]
classmethod of(compress_types, stored_lens, *, data_start)

The index of chunks laid out back to back from data_start.

classmethod uncompressed(chunk_num)

The uncompressed layout’s index: every chunk CompressType.NONE, FIXED_CHUNK_LENGTH apart from RESERVED_LENG (FORMAT-SPEC.md: Header & mode bits). Current writers never produce it.

locator(chunk_idx)

Chunk chunk_idx’s absolute byte range.

property total_effective_len: int

Bytes of the whole chunk-data region.

property non_empty_count: int

Chunks with effective_len > 0, so COMPACTED ones count as empty. The spec does not distinguish this from counting chunk_num verbatim, and no real compacted chunk has settled it.

crc_store_positions()

Every chunk’s position within the ChunkCrcStore trailer: the count of non-empty chunks below it. A COMPACTED chunk’s slot holds the next non-empty chunk’s position and is never read.

synology_apm_repo.sdk.format.bucket.expected_bucket_size(header, index)

Self-check total on-disk file size.

COMPRESS_RESERVED_LENG + Σ effective_len + 4×non-empty-chunks + redundancy_size(tight_sizestore_len, 256).

Raises:

ValueError – header is not the compressed layout.

synology_apm_repo.sdk.format.bucket.chunk_crc_store_region(header, index)

Byte (offset, length) of the ChunkCrcStore trailer (FORMAT-SPEC.md: ChunkCrcStore & Redundancy) within the bucket file — immediately after the chunk-data region, one 4-byte entry per non-empty chunk.

Raises:

ValueError – header is not the compressed layout.

synology_apm_repo.sdk.format.bucket.parse_chunk_crc_store(data, non_empty, *, verify_crc=None)

Decode the ChunkCrcStore trailer: one big-endian 4-byte ciphertext CRC32 per non-empty chunk (FORMAT-SPEC.md: ChunkCrcStore & Redundancy), computed at write time over each chunk’s stored bytes (after compression and encryption, if any) — not a plaintext check.

Parameters:
  • data (bytes) – Bytes starting at chunk_crc_store_region’s offset; only the first CHUNK_CRC_SIZE * non_empty bytes are read.

  • non_empty (int) – Number of entries, from chunk_crc_store_region.

  • verify_crc (int | None) – The header’s crc_of_chunk_crc field. When given, the trailer’s own CRC32 is checked against it (no chunk’s data is checked here).

Raises: