synology_apm_repo.sdk.dedup.pool package¶
Chunk pool: resolves a ChunkAddress to plaintext bytes.
The only place chunk decrypt+decompress happens. Three collaborating
classes: BucketReader (_bucket_reader.py; one .buk file’s
header/SizeStore/locators plus the per-chunk read), Pool (this module;
the repository-wide entry point, which resolves (streamID, bucketID) to a
.buk path and caches BucketReaders and decoded chunks), and
BucketReaderCache (_cache.py; the private cache a bulk sweep uses
instead of Pool’s own).
- class synology_apm_repo.sdk.dedup.pool.BucketReader(store, path, header, index, vault_key, *, algorithm=None, verify_ciphertext_crc=False, sizestore_repaired=False, known_file_size=None)¶
Bases:
objectOne opened
.bukfile. Build withopen.The read paths use
index’s raw columns directly, since a bucket-major export touches nearly every chunk.- Variables:
path – The
.bukpath in the store.header – The parsed bucket header.
index – Every chunk’s SizeStore entry and byte range.
sizestore_repaired –
openfixed a SizeStore CRC mismatch via Redundancy parity (check_bucket_structurereports it).known_file_size – The on-disk size, when that repair had to fetch it.
- async classmethod open(store, path, *, vault_key=None, algorithm=None, verify_ciphertext_crc=False)¶
Open
pathand parse its header + SizeStore in one read of up toCOMPRESS_RESERVED_LENG(16384) bytes.The SizeStore CRC is always verified. A mismatch first tries the bucket’s trailing Redundancy blob (
_attempt_size_store_repair), so parity repair is transparent on every open; a successful repair setssizestore_repaired.- Parameters:
store (ObjectStore) – The object store holding
path.path (str) – Physical
.bukpath.vault_key (bytes | None) – Required to read chunks of a vault-encrypted bucket, even when a prebuilt AES
algorithmis passed.algorithm (AES | None) – A key schedule built from
vault_key, reused by every chunk decrypt instead of rebuilding it.verify_ciphertext_crc (bool) – Make every chunk read check its stored bytes against the ChunkCrcStore first; opening does not read the trailer itself.
- Raises:
NotFoundError –
pathdoes not exist.FormatError – The file is shorter than its header.
DataCorruptError – The header or SizeStore is corrupt and not repairable.
UnsupportedVersionError – The bucket’s major version is too new.
- async ensure_chunk_crc_store()¶
Lazily read and self-validate this bucket’s ChunkCrcStore trailer (FORMAT-SPEC.md: ChunkCrcStore & Redundancy), caching the per-chunk ciphertext CRC32 values.
- Returns:
The CRC32 values, one per stored chunk.
- Raises:
FormatError – The trailer is shorter than declared.
DataCorruptError – The trailer’s bytes don’t match the header’s
crcOfChunkCrcself-consistency field.
- Return type:
- async verify_chunk_ciphertext_crc(chunk_idx)¶
Check chunk
chunk_idx’s stored bytes against itsChunkCrcStoreentry without decrypting, so no vault key is needed.- Raises:
DataCorruptError – The ciphertext doesn’t match its entry.
ChunkCompactedError –
chunk_idxisCOMPACTED.
- async verify_raw_chunk_ciphertext_crc(chunk_idx, raw)¶
verify_chunk_ciphertext_crcfor a caller that already holdschunk_idx’s stored bytes (e.g. fromread_raw_chunks).- Raises:
DataCorruptError –
rawdoesn’t matchchunk_idx’s ownChunkCrcStoreentry.
- non_compacted_chunk_ranges()¶
Every chunk with readable data, as ascending
(chunk_idx_start, length)runs that skipCOMPACTEDslots (reclaimed, not corruption). Scans the raw compress-type bytes in C, not per chunk.
- async read_chunk(chunk_idx, addr)¶
Decrypt (if vault-encrypted) and decompress chunk
chunk_idx, first checking its stored bytes against itsChunkCrcStoreentry (FORMAT-SPEC.md: ChunkCrcStore & Redundancy) when opened withverify_ciphertext_crc(one trailer read per bucket).- Parameters:
chunk_idx (ChunkIdx) – The chunk within this bucket.
addr (ChunkAddress) – The chunk’s own
ChunkAddress, supplying the decryption IV.
- Returns:
Exactly 4096 bytes of plaintext.
- Raises:
ChunkCompactedError – The chunk is
COMPACTED.KeyRequiredError – The bucket is encrypted and no vault key was given.
DataCorruptError –
chunk_idxis past this bucket’s chunks, the CRC check fails or the chunk doesn’t decompress to 4096 bytes.
- Return type:
- async decode_raw_chunk(chunk_idx, addr, raw)¶
read_chunkfor a caller that already holdschunk_idx’s stored bytes (e.g. fromread_raw_chunks).
- async read_raw_chunk(chunk_idx)¶
Chunk
chunk_idx’s stored bytes exactly as written (still compressed and/or encrypted), for ChunkCrcStore checks (FORMAT-SPEC.md: ChunkCrcStore & Redundancy).- Raises:
DataCorruptError –
chunk_idxis past this bucket’s chunks.ChunkCompactedError –
chunk_idxisCOMPACTED.
- async read_raw_chunks(chunk_indices)¶
Batch form of
read_raw_chunk: locator byte-ranges within_GAP_TOLERANCEof each other share oneObjectStore.read.- Parameters:
chunk_indices (Sequence[int]) – Sorted ascending and deduplicated.
- Raises:
DataCorruptError – A chunk is past this bucket’s chunks.
ChunkCompactedError – A chunk is
COMPACTED; raised before any read.
- async read_chunks(stream_id, bucket_id, ranges)¶
Batch form of
read_chunk: locator byte-ranges within_GAP_TOLERANCEof each other share oneObjectStore.read, the merged reads issued one after another; decrypt is per chunk and decompress is batched per merged run (_decode_chunks/decompress_many).- Parameters:
- Returns:
Plaintext by
chunk_idx. A value may be amemoryviewinto a buffer it shares with other chunks of its run; holding it keeps that buffer alive (copy withbytes()to keep one chunk).- Raises:
ChunkCompactedError – A requested chunk is
COMPACTED; raised before any read.KeyRequiredError – The bucket is encrypted and no vault key was given.
DataCorruptError – A range runs past this bucket’s chunks, a CRC check fails or a chunk doesn’t decompress to 4096 bytes.
- Return type:
dict[int, bytes | memoryview]
- async read_chunk_blocks(stream_id, bucket_id, ranges, on_block, *, semaphore=None)¶
read_chunks’ merged reads, each read’s chunks decoded and handed toon_blockoneDECODE_BLOCK_CHUNKS-aligned block at a time and dropped once it returns. A block’s chunks can arrive in several calls, one per merged read covering them; calls for different reads may run concurrently.- Parameters:
stream_id (StreamId) – This bucket’s stream, for an encrypted chunk’s IV.
bucket_id (BucketId) – This bucket’s id, likewise.
on_block (Callable[[list[tuple[int, int]], dict[int, bytes | memoryview]], Awaitable[None]]) – Receives each block; a chunk may be a view into its read’s buffer, as
read_chunksdescribes.semaphore (Semaphore | None) – Shared read-concurrency bound (
None: serial), the oneexec_chunks()draws from. The caller has already acquired one permit for this call, which reads its merged runs in turn; a further run is read alongside only on a permit that is free when the call starts, never one it waits for. A permit is held while its read’s blocks are decoded and handed over.
- Raises:
ChunkCompactedError – A requested chunk is
COMPACTED; raised before any read.KeyRequiredError – The bucket is encrypted and no vault key was given.
DataCorruptError – A range runs past this bucket’s chunks, a CRC check fails or a chunk doesn’t decompress to 4096 bytes.
- class synology_apm_repo.sdk.dedup.pool.BucketReaderCache(maxsize=16)¶
Bases:
AsyncKeyedCache[tuple[StreamId,BucketId],BucketReader]Bucket-major export uses one instead of
Pool’s bounded cache, so a sweep touching every bucket once doesn’t evict hot interactive entries. Read throughPool.bucket(..., cache=); never shared withPool’s own cache. One instance spans every fragment of aVirtualDiskContentSourceexport; a standaloneexport_rangegets its own.Decoded chunk plaintext is not cached here: it stays scoped to one bucket-group call, which lets
decompress_manyreturn zero-copymemoryviewvalues.
- class synology_apm_repo.sdk.dedup.pool.Pool(store, pool_root, dir_cache, *, vault_key=None, limits=CacheLimits(bucket_readers=16, bucket_readers_interactive=64, chunks=4096, composition_records=64, composition_pages=128, saas_streams=8, dir_scan=256, dir_fixed=64, allocation_tables=256, forward_resolutions=4096), verify=VerifyPolicy(ciphertext_crc=False, fingerprint=False))¶
Bases:
objectRepository-wide chunk pool entry point.
Resolves any
ChunkAddressto its 4096-byte plaintext, with two-level LRU caching:BucketReaders (cheap, kept many) and decoded plaintext chunks (kept fewer). Both are sharedAsyncKeyedCacheinstances, so concurrent callers missing the same key share one fetch;backfill_chunksis the one direct insert that bypasses that, so its callers takebatch_fetchinstead.- Parameters:
store (ObjectStore) – The repository’s
ObjectStore.pool_root (str) – Path of the
@data/Pooldirectory.dir_cache (DirCache) – Shared
DirCachefor.<seqId>resolution.vault_key (bytes | None) – The VaultKey for an encrypted repository, else
None.limits (CacheLimits) – Bounds of the bucket-reader (
bucket_readers), plaintext chunk (chunks) and allocation-table (allocation_tables) caches.verify (VerifyPolicy) – The per-chunk checks every read runs.
- release_caches()¶
Drop everything this pool holds in memory: decoded chunks, open bucket readers, and allocation tables.
DirCacheis untouched; it belongs to the store.
- cache_stats()¶
Counters of this pool’s three caches, keyed
pool.buckets,pool.chunksandpool.allocation_tables.
- property release_epoch: int¶
Bumped once per
release_caches()call. A caller fetching plaintext outsideread_chunk()(dedup_file.py’s_resolve_bucket_group) captures it first and skipsbackfill_chunksif it changed.
- property store: ObjectStore¶
The
ObjectStorethis pool reads from (used byPoolDescriptor.from_pool).
- property limits: CacheLimits¶
The bounds this pool’s caches were built with.
- property verify: VerifyPolicy¶
The per-chunk checks every read through this pool runs.
- async bucket_path(stream_id, bucket_id)¶
Resolve
(stream_id, bucket_id)to its store-relative physical.bukpath, including any.<seqId>generation suffix.- Raises:
NotFoundError – No such bucket file.
- async bucket_size(stream_id, bucket_id)¶
The on-disk size of
bucket_path(stream_id, bucket_id)as its directory listing reported it, with nosizerequest;Noneif the backend lists no sizes (store.sizeis then the way to ask). The listing is cached, so a store that changed after it was read needsRepository.invalidate_caches().- Raises:
NotFoundError – No such bucket file.
- async bucket(stream_id, bucket_id, *, cache=None)¶
The
BucketReaderfor(stream_id, bucket_id), opened on first access and cached incache— a bulk sweep’s private one — or, whenNone, in this pool’s own.
- async open_bucket_uncached(stream_id, bucket_id)¶
Open
(stream_id, bucket_id)fresh, without touching this pool’s bucket cache (no lookup, insert or eviction).Every
bucket()cache miss, this pool’s own cache or a sweep’sBucketReaderCache, opens through it, as does a verify run’s bucket check.
- async read_chunk(addr)¶
Resolve
addrto 4096 bytes of plaintext, cached, with this pool’sverifychecks (a cache hit’s fingerprint is checked too). Bucket-major export and verify FULL read throughchunk_walk.read_verified_blocksinstead.- Returns:
The plaintext chunk.
- Raises:
NotFoundError – The bucket file is missing.
ChunkCompactedError – The chunk was reclaimed by compaction.
DataCorruptError – The bucket fails to decode, or a fingerprint or CRC check fails.
- Return type:
- cached_chunks(stream_id, bucket_id, chunk_indices)¶
(hits, misses)forchunk_indicesof one bucket: the plaintext already cached, by index, and the indices that are not; never fetches. Each lookup counts and refreshes recency asread_chunk()’s would. For batch callers (dedup_file.py) that skipread_chunk().
- batch_fetch(stream_id, bucket_id)¶
Held around a batch caller’s
cached_chunks-fetch-backfill_chunksof one bucket, so concurrent batches of the same bucket run one at a time and a later one finds what an earlier one cached instead of fetching it again.
- backfill_chunks(stream_id, bucket_id, plaintexts)¶
Cache decoded
plaintexts(bychunk_idx) of one bucket as ifread_chunk()had fetched them; amemoryviewis copied, so the cache never pins the buffer it views. Verifies no fingerprint; that is the caller’s job.
- async verify_fingerprints(stream_id, bucket_id, chunks)¶
Check every entry in
chunks(decoded plaintext keyed bychunk_idx) against its stored.fgpdigest when this pool’sverify.fingerprintis on:read_chunk’s policy, for callers ofBucketReader.read_chunks.- Parameters:
- Raises:
DataCorruptError – Any chunk’s plaintext SHA-256 doesn’t match its stored fingerprint, or a sidecar is corrupt.
NotFoundError – The group’s
.infor.fgpfile is missing.FormatError – The
.infor.fgpfile is truncated.
- async expected_fingerprints(stream_id, bucket_id, chunk_indices)¶
The stored
.fgpdigest of everychunk_idxinchunk_indices, forcheck_fingerprints;Nonewhen this pool’sverify.fingerprintis off. Consecutive indices share one read.- Raises:
DataCorruptError – A sidecar is corrupt.
NotFoundError – The group’s
.infor.fgpfile is missing.FormatError – The
.infor.fgpfile is truncated.
- check_fingerprints(stream_id, bucket_id, chunks, expected)¶
Check every entry in
chunksagainst its digest inexpected(fromexpected_fingerprints), with no I/O.- Raises:
DataCorruptError – A chunk’s plaintext SHA-256 doesn’t match.
- class synology_apm_repo.sdk.dedup.pool.VerifyPolicy(ciphertext_crc=False, fingerprint=False)¶
Bases:
objectThe per-chunk checks every read through a
Poolruns. Browsing and export run neither (NO_VERIFY);VerifyLevel.FULLruns both (FULL_VERIFY) — seeARCHITECTURE.md’s validation table.