synology_apm_repo.sdk.format.bucket module

Bucket file (.buk) format (FORMAT-SPEC.md §4).

Pure bytes -> dataclass decode, zero I/O — this module never opens a file. The Dedup Layer’s BucketReader is what actually fetches bytes from an ObjectStore at the offsets this module computes, then hands ciphertext through format.crypto and compression.

ABP builds always write mode = COMPRESS|CHUNK_CRC (0x03), optionally with VAULT_ENCRYPT (0x83) — but every mode bit is checked explicitly here rather than assumed: FORMAT-SPEC.md: bucket-header warns that a bit always being 1 today is an observation, not a guarantee the check can be skipped.

synology_apm_repo.sdk.format.bucket.COMPRESS_TYPE_BY_VALUE = {0: CompressType.NONE, 1: CompressType.LZ4, 2: CompressType.ZSTD, 4: CompressType.COMPACTED}

Plain-dict stand-in for CompressType(value), used in parse_size_store’s per-chunk hot loop (up to 8192 calls per bucket) in place of Enum.__call__’s own value-lookup machinery. Every other CompressType(...) call site in this codebase is cold (once per chunk at most, not in a hot per-bucket loop) and is left as the plain, self-explaining enum call.

synology_apm_repo.sdk.format.bucket.COMPRESS_TYPE_COMPACTED_VALUE = 4

Raw value compared directly in _SizeStoreArray.effective_lens’s own hot loop, same reasoning as COMPRESS_TYPE_BY_VALUE above, one level further — that loop never needs the CompressType member itself, only to compare against these two particular values.

class synology_apm_repo.sdk.format.bucket.BucketFileHeader(major, minor, mode, chunk_num, chunk_size_crc, crc_of_chunk_crc)

Bases: object

Parsed .buk header (FORMAT-SPEC.md: bucket-header).

major: int
minor: int
mode: int
chunk_num: int
chunk_size_crc: int
crc_of_chunk_crc: int

CRC32 over the whole ChunkCrcStore trailer (FORMAT-SPEC.md: ChunkCrcStore) — not needed to read a chunk’s content; a verify caller checks it against chunk_crc_store_region’s bytes.

property is_compressed: bool
property is_vault_encrypted: bool

The only reliable signal that this bucket’s chunks are encrypted — never repo_info’s encrypt_algorithm, which is a compile-time constant on ABP builds and unrelated to any individual bucket’s actual state.

synology_apm_repo.sdk.format.bucket.parse_bucket_header(data)

Parse a .buk file’s 64-byte header from the start of data.

class synology_apm_repo.sdk.format.bucket.SizeStoreEntry(compress_type, stored_len)

Bases: NamedTuple

One chunk’s compression type and stored (compressed) length, as recorded in SizeStore (FORMAT-SPEC.md: SizeStore).

NamedTuple, not this project’s usual @dataclass(frozen=True) — a deliberate exception: parse_size_store constructs up to 8192 of these per bucket, millions in aggregate across a real export, and a plain NamedTuple build has identical attribute access with no introspection callers to break, at lower construction cost than @dataclass. parse_size_store hands these out lazily via _SizeStoreArray rather than building every entry up front.

compress_type: CompressType

Alias for field number 0

stored_len: int

The raw 12-bit size field. Not the same as effective_len.

property effective_len: int

Actual bytes this chunk occupies in the data region. 4096 for CompressType.NONE (stored_len is meaningless there), 0 for CompressType.COMPACTED, otherwise stored_len verbatim.

synology_apm_repo.sdk.format.bucket.chunk_size_store_tight_length(chunk_num)

ceil(chunk_num * 15 / 8), the tightly-packed SizeStore length in bytes, before zero-padding to the fixed 16320-byte on-disk allocation.

synology_apm_repo.sdk.format.bucket.parse_size_store(data, chunk_num, *, verify_crc=None)

Decode chunk_num 15-bit packed SizeStore records starting at the beginning of data (FORMAT-SPEC.md: SizeStore, the bucket’s SizeStore region, bytes [64, 16384)); only the first chunk_size_store_tight_length bytes plus a little slack are actually read.

Decodes _SIZE_STORE_GROUP_RECORDS (8) records at a time (see that constant for the bit-packing this relies on). Each record’s decoded compress_type/stored_len still gets eager validation (raising DataCorruptError immediately for an unknown CompressType) — only building a SizeStoreEntry object is deferred, to _SizeStoreArray’s own __getitem__.

Parameters:

  • data (bytes) – Bytes starting at the SizeStore region.

  • chunk_num (int) – Number of records to decode, from the bucket header.

  • verify_crc (int | None) – The header’s chunkSizeCrc field. When given, the tightly-packed bytes’ CRC32 is checked against it — every open validates this.

Raises:

  • FormatError – data is shorter than the tightly-packed region chunk_num implies.

  • DataCorruptError – verify_crc was given and does not match, or a decoded record’s CompressType value is unknown.

class synology_apm_repo.sdk.format.bucket.ChunkLocator(offset, length)

Bases: NamedTuple

Absolute file byte-range for one chunk’s (still compressed and/or encrypted) data.

Same deliberate NamedTuple-not-@dataclass(frozen=True) exception as SizeStoreEntry above — see there for the reasoning.

chunk_locators hands these out lazily too, via _ChunkLocatorArray, when it’s fed a _SizeStoreArray: a real ChunkLocator is built only for the index actually asked for.

offset: int

Alias for field number 0

length: int

Alias for field number 1

synology_apm_repo.sdk.format.bucket.chunk_locators(header, entries)

Absolute file byte-range for every chunk described by entries (FORMAT-SPEC.md: bucket-physical-layout).

ABP builds always use the compressed layout (data starts at COMPRESS_RESERVED_LENG, 16384); the uncompressed layout (fixed 4096 bytes/chunk starting at RESERVED_LENG, 4096) is handled too, but is never produced by any current writer.

synology_apm_repo.sdk.format.bucket.raw_chunk_arrays(header, entries, locators)

(compress_type_values, offsets, lengths) as three flat array.array buffers — O(1) per-chunk access to every field SizeStoreEntry/ChunkLocator carry without constructing either object, for a caller whose access pattern is dense enough (every chunk in every bucket it opens) that even lazy construction adds up.

Zero-copy whenever header.is_compressed — every current writer’s layout — returning the exact arrays parse_size_store/ chunk_locators already built; the assert below is a fail-fast check that entries/locators actually came from that branch. The uncompressed layout instead gets a formulaic fill (every chunk is CompressType.NONE at a fixed FIXED_CHUNK_LENGTH stride from RESERVED_LENG), with nothing to read from entries/locators at all.

synology_apm_repo.sdk.format.bucket.expected_bucket_size(header, entries)

Self-check total on-disk file size.

COMPRESS_RESERVED_LENG + Σ effective_len + 4×non-empty-chunks + redundancy_size(tight_sizestore_len, 256).

Raises:

ValueError – header is not the compressed (ABP) layout.

synology_apm_repo.sdk.format.bucket.chunk_crc_store_region(header, entries)

Byte (offset, length) of the ChunkCrcStore trailer (FORMAT-SPEC.md: ChunkCrcStore) within the bucket file — immediately after the chunk-data region, one 4-byte entry per non-empty chunk.

Raises:

ValueError – header is not the compressed (ABP) layout.

synology_apm_repo.sdk.format.bucket.chunk_crc_store_index(entries, chunk_idx)

Position of chunk chunk_idx within the ChunkCrcStore trailer — the count of non-empty (effective_len > 0) chunks at indices below it, since a COMPACTED chunk has no trailer entry of its own at all.

O(chunk_idx) per call — correct but quadratic if called once per chunk for every chunk in a bucket, since a real bucket can hold thousands of chunks. A one-off single-chunk lookup (the only current use, verify’s spot-checks) is fine; a caller resolving every chunk in a bucket wants chunk_crc_store_positions instead.

synology_apm_repo.sdk.format.bucket.chunk_crc_store_positions(entries)

Every chunk’s own ChunkCrcStore position, computed in one O(n) cumulative-count pass — the batch counterpart to chunk_crc_store_index’s O(1)-but-called-once-per-chunk-is-O(n²) single lookup. A COMPACTED chunk’s own slot holds whatever position the next non-empty chunk would get (it has no real entry of its own; never read for one).

synology_apm_repo.sdk.format.bucket.parse_chunk_crc_store(data, non_empty, *, verify_crc=None)

Decode the ChunkCrcStore trailer: one big-endian 4-byte ciphertext CRC32 per non-empty chunk (FORMAT-SPEC.md: ChunkCrcStore), computed at write time over each chunk’s stored bytes (after compression and encryption, if any) — not a plaintext check.

Parameters:

  • data (bytes) – Bytes starting at chunk_crc_store_region’s offset; only the first CHUNK_CRC_SIZE * non_empty bytes are read.

  • non_empty (int) – Number of entries, from chunk_crc_store_region.

  • verify_crc (int | None) – The header’s crc_of_chunk_crc field. When given, the whole trailer’s own CRC32 is checked against it — this is a self-consistency check of the trailer bytes, not a check of any individual chunk’s stored data against its own entry here.

Raises: