synology_apm_repo.sdk.format.bucket module¶
Bucket file (.buk) format (FORMAT-SPEC.md §4).
Pure bytes -> dataclass decode, zero I/O. The Dedup Layer’s
BucketReader fetches bytes at the offsets this module computes, then
hands ciphertext through format.crypto and compression.
In practice mode is COMPRESS|CHUNK_CRC (0x03), plus
VAULT_ENCRYPT (0x83) for an encrypted bucket, but readers test each
bit rather than assume it (FORMAT-SPEC.md: Header & mode bits).
- synology_apm_repo.sdk.format.bucket.COMPRESS_TYPE_BY_VALUE = {0: CompressType.NONE, 1: CompressType.LZ4, 2: CompressType.ZSTD, 4: CompressType.COMPACTED}¶
Plain-dict stand-in for
CompressType(value)in the per-chunk hot loops (up to 8192 calls per bucket), avoidingEnum.__call__overhead.
- synology_apm_repo.sdk.format.bucket.COMPRESS_TYPE_COMPACTED_VALUE = 4¶
CompressType.COMPACTED’s raw value, for hot loops that compare raw SizeStore values (BucketIndex,BucketReader) without buildingCompressTypemembers.
- class synology_apm_repo.sdk.format.bucket.BucketFileHeader(major, minor, mode, chunk_num, chunk_size_crc, crc_of_chunk_crc)¶
Bases:
objectParsed
.bukheader (FORMAT-SPEC.md: Header & mode bits).
- synology_apm_repo.sdk.format.bucket.parse_bucket_header(data)¶
Parse a
.bukfile’s 64-byte header from the start ofdata.- Raises:
FormatError –
datais shorter than 64 bytes.DataCorruptError – A magic or header-CRC mismatch.
UnsupportedVersionError –
majoris above 3.
- class synology_apm_repo.sdk.format.bucket.SizeStoreEntry(compress_type, stored_len)¶
Bases:
NamedTupleOne chunk’s compression type and stored (compressed) length, as recorded in
SizeStore(FORMAT-SPEC.md: SizeStore).A
NamedTuple, not a frozen dataclass: built up to 8192 times per bucket, where it is cheaper.- compress_type: CompressType¶
Alias for field number 0
- synology_apm_repo.sdk.format.bucket.chunk_size_store_tight_length(chunk_num)¶
ceil(chunk_num * 15 / 8), the tightly-packed SizeStore length in bytes, before zero-padding to the fixed 16320-byte on-disk allocation.
- synology_apm_repo.sdk.format.bucket.parse_size_store(data, chunk_num, *, verify_crc=None)¶
Decode
chunk_num15-bit packed SizeStore records starting at the beginning ofdata(FORMAT-SPEC.md: SizeStore, the bucket’s SizeStore region, bytes[64, 16384)); only the firstchunk_size_store_tight_lengthbytes are read.Every record’s
CompressTypeis validated eagerly; the result is the compressed layout’sBucketIndex.- Parameters:
- Raises:
FormatError –
datais shorter than the tightly-packed regionchunk_numimplies.DataCorruptError –
verify_crcwas given and does not match, or a decoded record’sCompressTypevalue is unknown.
- class synology_apm_repo.sdk.format.bucket.ChunkLocator(offset, length)¶
Bases:
NamedTupleAbsolute file byte-range for one chunk’s (still compressed and/or encrypted) data. A
NamedTuple, not a frozen dataclass: built per chunk read, where it is cheaper.
- class synology_apm_repo.sdk.format.bucket.BucketIndex(compress_types, stored_lens, effective_lens, offsets)¶
Bases:
Sequence[SizeStoreEntry]One bucket’s per-chunk SizeStore entries and data locations, as parallel
array.arraycolumns (FORMAT-SPEC.md: SizeStore, Physical layout). ASequence[SizeStoreEntry];locator(i)is chunki’s byte range. Hot loops read the columns directly; they are shared, so never mutate them.- Variables:
compress_types (array.array[int]) – Raw
CompressTypevalues ("B").stored_lens (array.array[int]) – Raw 12-bit size fields (
"H").effective_lens (array.array[int]) – Bytes each chunk occupies in the data region (
SizeStoreEntry.effective_len;"H"); also each locator’s length.offsets (array.array[int]) – Each chunk’s absolute file offset (
"I": a bucket tops out at ~33.6 MB).
- classmethod of(compress_types, stored_lens, *, data_start)¶
The index of chunks laid out back to back from
data_start.
- classmethod uncompressed(chunk_num)¶
The uncompressed layout’s index: every chunk
CompressType.NONE,FIXED_CHUNK_LENGTHapart fromRESERVED_LENG(FORMAT-SPEC.md: Header & mode bits). Current writers never produce it.
- locator(chunk_idx)¶
Chunk
chunk_idx’s absolute byte range.
- property non_empty_count: int¶
Chunks with
effective_len > 0, soCOMPACTEDones count as empty. The spec does not distinguish this from countingchunk_numverbatim, and no real compacted chunk has settled it.
- crc_store_positions()¶
Every chunk’s position within the ChunkCrcStore trailer: the count of non-empty chunks below it. A COMPACTED chunk’s slot holds the next non-empty chunk’s position and is never read.
- synology_apm_repo.sdk.format.bucket.expected_bucket_size(header, index)¶
Self-check total on-disk file size.
COMPRESS_RESERVED_LENG + Σ effective_len + 4×non-empty-chunks + redundancy_size(tight_sizestore_len, 256).- Raises:
ValueError –
headeris not the compressed layout.
- synology_apm_repo.sdk.format.bucket.chunk_crc_store_region(header, index)¶
Byte
(offset, length)of the ChunkCrcStore trailer (FORMAT-SPEC.md: ChunkCrcStore & Redundancy) within the bucket file — immediately after the chunk-data region, one 4-byte entry per non-empty chunk.- Raises:
ValueError –
headeris not the compressed layout.
- synology_apm_repo.sdk.format.bucket.parse_chunk_crc_store(data, non_empty, *, verify_crc=None)¶
Decode the ChunkCrcStore trailer: one big-endian 4-byte ciphertext CRC32 per non-empty chunk (FORMAT-SPEC.md: ChunkCrcStore & Redundancy), computed at write time over each chunk’s stored bytes (after compression and encryption, if any) — not a plaintext check.
- Parameters:
data (bytes) – Bytes starting at
chunk_crc_store_region’s offset; only the firstCHUNK_CRC_SIZE * non_emptybytes are read.non_empty (int) – Number of entries, from
chunk_crc_store_region.verify_crc (int | None) – The header’s
crc_of_chunk_crcfield. When given, the trailer’s own CRC32 is checked against it (no chunk’s data is checked here).
- Raises:
FormatError –
datais shorter than the trailer needs.DataCorruptError –
verify_crcwas given and does not match.