synology_apm_repo.sdk.dedup.pool package

Chunk pool: resolves a ChunkAddress to plaintext bytes.

The only place chunk decrypt+decompress happens. Three collaborating classes: BucketReader (_bucket_reader.py — one .buk file’s header/SizeStore/locators plus the actual per-chunk read), Pool (this module — repository-wide entry point — resolves (streamID, bucketID) to the right .buk path and caches both BucketReaders and decoded plaintext chunks), and BucketReaderCache (_cache.py — the private cache shape a bulk sweep needs instead of Pool’s own session-wide bounded one — unbounded by default, since export’s own repeat-visit pattern needs it, but a caller whose own access pattern doesn’t benefit from unbounded growth passes its own maxsize).

synology_apm_repo.sdk.dedup.pool.DEFAULT_BUCKET_CACHE_SIZE = 16

Pool’s own default bucket_cache_size – also the value every BucketReaderCache(maxsize=...) construction site outside this module (chunk_walk.py, export_scheduler.py, units.content.pcps_disk) reuses for its own default bound, so those three stay tied to this one number instead of each separately hardcoding a copy that could silently drift from it.

synology_apm_repo.sdk.dedup.pool.DEFAULT_CHUNK_CACHE_SIZE = 4096

Pool’s own default chunk_cache_size – also reused by dedup.repository.DedupRepo’s own matching default, so the two stay tied to this one number instead of each separately hardcoding a copy that could silently drift from it.

synology_apm_repo.sdk.dedup.pool.INTERACTIVE_BUCKET_CACHE_SIZE = 64

bucket_cache_size for the one Pool every non-bulk consumer of a repository’s own catalog shares (api.repository.Repository. _open_catalog_resources, reached by both TUI/SDK browsing and one-shot CLI commands like ls/tree/doctor via Repository.catalogs()) – bigger than DEFAULT_BUCKET_CACHE_SIZE because a single real directory listing under a PC/VM device’s disk image can touch several dozen distinct .buk files at once, which DEFAULT_BUCKET_CACHE_SIZE’s 16 slots can’t hold without evicting and later re-opening some of them mid-listing. Deliberately a new constant rather than raising DEFAULT_BUCKET_CACHE_SIZE itself, which bulk-export’s own BucketReaderCaches are tuned to for a different access pattern (each bucket touched once, sequentially across a whole sweep) that gains nothing from a bigger bound the way one deep tree walk does.

class synology_apm_repo.sdk.dedup.pool.BucketReader(store, path, header, entries, vault_key, *, default_verify_ciphertext_crc=False, sizestore_repaired=False, known_file_size=None)

Bases: object

entries/locators (lazy Sequences) are for external, sparse-access callers that only ever touch a handful of a bucket’s chunks (verify_checks.py’s spot-checks, the dump CLI command). This class’s own per-chunk reads (read_chunk, read_chunks) go straight to the raw array.array values raw_chunk_arrays returns instead, skipping per-chunk SizeStoreEntry/ChunkLocator construction entirely — a bucket-major export touches nearly every chunk in nearly every bucket it opens, dense enough that even lazy construction adds up.

async classmethod open(store, path, *, vault_key=None, verify_ciphertext_crc=False)

Open path and parse its header + SizeStore in one read of up to COMPRESS_RESERVED_LENG (16384) bytes.

SizeStore’s CRC is always verified here (this is not configurable to skip — you cannot compute chunk locators without first reading the SizeStore bytes, so validating them is free at that point, unlike the composition layer’s much more expensive full-map-array mapCrc). A CRC mismatch is not immediately fatal, though: this first tries the bucket’s own trailing Redundancy blob (see _attempt_size_store_repair) — every current-format bucket carries one (FORMAT-SPEC.md: ChunkCrcStore & Redundancy) — since this check does run on every open, unconditionally, this is the one place in the whole SDK where parity self-repair is genuinely transparent for every caller, not just an explicit verify/diagnostics check. A successful repair sets the returned reader’s own sizestore_repaired — this method itself has no Finding-returning contract, so verify_checks. check_bucket_structure is what actually reports it.

verify_ciphertext_crc becomes this reader’s own read_chunk/read_chunks default — it does not itself trigger reading the ChunkCrcStore trailer here.

async ensure_chunk_crc_store()

Lazily read and self-validate this bucket’s ChunkCrcStore trailer (FORMAT-SPEC.md: ChunkCrcStore), caching the per-chunk ciphertext CRC32 values for read_chunk/read_chunks’s own verify_ciphertext_crc option — and for a caller doing its own one-off structural self-consistency check — to share without a second read of the same bytes.

Raises:

• FormatError – The trailer is shorter than declared.

• DataCorruptError – The trailer’s own bytes don’t match the header’s crcOfChunkCrc self-consistency field.

async verify_chunk_ciphertext_crc(chunk_idx)

Check chunk chunk_idx’s stored bytes against its ChunkCrcStore entry, with no decrypt/decompress attempt at all — unlike read_chunk’s own verify_ciphertext_crc= option (which still goes on to decrypt afterward, and so still needs a vault key for an encrypted bucket), this works whether or not a key is even available. For a caller (verify) that wants this one check in isolation, independent of KeyRequiredError.

Raises:

DataCorruptError – The chunk’s ciphertext doesn’t match its ChunkCrcStore entry.

async verify_raw_chunk_ciphertext_crc(chunk_idx, raw)

The second half of verify_chunk_ciphertext_crc, split out for a caller that already has chunk_idx’s stored bytes in hand — e.g. from a batched read_raw_chunks call — instead of fetching them itself.

Raises:

DataCorruptError – raw doesn’t match chunk_idx’s own ChunkCrcStore entry.

non_compacted_chunk_indices()

Every chunk index with real, readable data — skips COMPACTED slots (reclaimed, not corruption). Built from the raw compress-type array, the same source read_chunk/read_chunks use for their own per-chunk decoding, so a whole-bucket sweep costs one O(chunk_num) pass over plain ints with no per-chunk SizeStoreEntry construction — unlike a caller wanting just one representative chunk instead, which should index entries directly rather than call this for a single lookup.

async read_chunk(chunk_idx, addr, *, verify_ciphertext_crc=None)

Decrypt (if the bucket is vault-encrypted) and decompress chunk chunk_idx, returning exactly 4096 bytes of plaintext.

addr supplies the IV for decryption — pass the chunk’s own ChunkAddress.

Reads straight off the raw arrays rather than entries/locators — no reason for even a single cold-path SizeStoreEntry/ChunkLocator construction once the raw arrays already exist from __init__.

verify_ciphertext_crc (None: defer to whatever this reader was open()ed with) compares this chunk’s stored bytes against its ChunkCrcStore entry (FORMAT-SPEC.md: ChunkCrcStore) before decrypting — a general data-integrity check independent of encryption and needing no vault key, off by default since it costs an extra trailer read the first time it’s used per bucket.

async decode_raw_chunk(chunk_idx, addr, raw, *, verify_ciphertext_crc=None)

The decrypt/decompress half of read_chunk, split out for a caller that already has chunk_idx’s stored bytes in hand — e.g. from a batched read_raw_chunks call — instead of fetching them itself, the same split verify_raw_chunk_ciphertext_crc already has from verify_chunk_ciphertext_crc.

verify_ciphertext_crc: same meaning as read_chunk’s own parameter. A caller that already ran verify_raw_chunk_ciphertext_crc on raw itself should pass False here — this would otherwise redo (and raise past) that same check.

async read_raw_chunk(chunk_idx)

Fetch chunk chunk_idx’s stored bytes exactly as written — compressed and/or encrypted per mode, no decrypt/decompress applied. For verify_checks.py’s ChunkCrcStore spot-check (FORMAT-SPEC.md: ChunkCrcStore), which checks the ciphertext CRC32 directly and has no need for the plaintext read_chunk produces.

Raises:

ChunkCompactedError – chunk_idx is COMPACTED — checked first (like read_chunk’s own _resolve_compress_type call), since a compacted chunk has no ChunkCrcStore entry of its own; a caller resolving one anyway would otherwise index one past the end of that trailer’s own array.

async read_raw_chunks(chunk_indices)

Batch form of read_raw_chunk: merges chunk_indices’ own locator byte-ranges that are within _GAP_TOLERANCE of each other into a single ObjectStore.read call instead of issuing one read per chunk — the same run-merging read_chunks does, minus the decrypt/decompress step neither this method nor the single-chunk read_raw_chunk performs. Collapsing the read count like this matters most on a remote-object-storage backend, where every separate read is its own network round trip.

Parameters:

chunk_indices (Sequence[int]) – Must already be sorted ascending and deduplicated — same contract as read_chunks’ requests.

Raises:

ChunkCompactedError – For the first COMPACTED chunk found in chunk_indices, raised up front before any read happens — same as read_chunks.

async read_chunks(requests, *, semaphore=None, verify_ciphertext_crc=None)

Batch form of read_chunk: merges requests’ locator byte-ranges that are within _GAP_TOLERANCE of each other into a single ObjectStore.read call instead of issuing one read per chunk, then decrypts/decompresses each chunk out of its own slice of whichever merged buffer it landed in. Decrypt stays strictly per-chunk; decompress is batched across a whole merged run instead, via _decode_run/decompress_many.

Parameters:

• requests (Sequence[tuple[int, ChunkAddress | None]]) – Must already be sorted by chunk_idx ascending and deduplicated — not re-sorted or re-deduplicated here, so a caller bug in either regard surfaces as a wrong/inefficient merge rather than being silently hidden. Each addr may be None when the caller already knows this bucket isn’t vault-encrypted, since _decode_run never dereferences it unless it is; a caller that doesn’t know ahead of time may still pass a real one.

• semaphore (Semaphore | None) – The shared concurrency pool for this whole export (default None: serial) — the same asyncio.Semaphore exec_chunks()’s cross-bucket dispatch loop also draws from, so total concurrently in-flight reads across both levels never exceed one shared bound. A caller passing a non-None semaphore has already acquired one permit on this call’s behalf; this method’s first merged run spends that permit directly, and only a second/third/… run (a bucket whose requested chunks land in more than one physically-separate region of its .buk file) acquires its own additional permit from the same pool.

• verify_ciphertext_crc (bool | None) – Same meaning as read_chunk’s own parameter (None defers to whatever this reader was open()ed with) — checked per chunk, before decrypting.

Returns:

A chunk’s value may be a memoryview rather than an independent bytes object — it can view into either decompress_many’s shared per-run decode buffer (ZSTD) or this method’s own merged I/O-read buffer (unencrypted NONE/ LZ4), never copied just to satisfy this method’s own contract. Every consumer today only reads the bytes forward into another buffer, never holding the value past its own call.

Raises:

ChunkCompactedError – For the first COMPACTED chunk found in requests, raised up front before any read happens.

Return type:

dict[int, bytes | memoryview]

class synology_apm_repo.sdk.dedup.pool.BucketReaderCache(maxsize=None)

Bases: object

Export’s bucket-major path and verify’s Bucket-and-key stage each build one instead of going through Pool’s own bounded _buckets, so a sweep touching every bucket once doesn’t evict genuinely-hot interactive entries from that shared cache. Reads/writes never cross over with Pool’s own cache — the two stay fully independent.

maxsize (None, the default: unbounded) is bounded by however many distinct buckets one sweep actually touches unless the caller passes a smaller cap — see each construction site’s own reasoning for why it picked the value it did. Built on AsyncKeyedCache like every other cache in this SDK, with fetch supplied per call (typically Pool.open_bucket_uncached) since this class is constructed at layers with no Pool reference of their own.

One instance is shared across every fragment of a VirtualDiskContentSource export, so a bucket one fragment opens stays open for the next; a standalone export_to call or a verify run instead each get their own separate instance.

Note

Deliberately does not also cache decoded chunk plaintext across calls — same-call repeats are already deduplicated by exec_chunks itself, and unconditionally remembering every decoded chunk would cost memory roughly equal to the whole export’s unique DATA content for little cross-call reuse against real VM-image exports. Chunk decode stays scoped to one bucket-group call, which is also what lets decompress_many hand back zero-copy memoryview values instead of an independent bytes copy per chunk.

maxsize: int | None = None

buckets: AsyncKeyedCache[tuple[StreamId, BucketId], BucketReader]

class synology_apm_repo.sdk.dedup.pool.Pool(store, pool_root, dir_cache, *, vault_key=None, bucket_cache_size=16, chunk_cache_size=4096, verify_fingerprint=False, verify_ciphertext_crc=False)

Bases: object

Repository-wide chunk pool entry point.

Resolves any ChunkAddress to its 4096-byte plaintext, with two-level LRU caching: BucketReaders (header + locators, cheap, kept many) and decoded plaintext chunks (more expensive, kept fewer). Both are AsyncKeyedCache instances — session-wide, shared across every interactive caller — which already gives concurrency-safety (an internal lock, held only around bookkeeping, never around the I/O of opening a bucket or reading/decrypting/decompressing a chunk) and in-flight fetch de-duplication (two concurrent callers missing the same key pay for one fetch, not two) for free. backfill_chunk is the one direct-insert exception to that de-duplication guarantee.

release_caches()

Drop everything this pool holds in memory: decoded chunks, open bucket readers, and allocation tables.

Closing a repository does not, on its own, free any of this — the caches live on the Pool, which a caller can still be holding a reference to. Anything walking several repositories in one process (a smoke run, a TUI session browsing one after another) would otherwise keep every pool it ever opened fully populated. The DirCache is deliberately untouched: it belongs to the store, not to this pool.

property release_epoch: int

Bumped once per release_caches() call – a caller fetching a chunk’s plaintext some other way than read_chunk()/resolve() (dedup_file.py’s _resolve_bucket_group) captures this before starting, then skips backfill_chunk if it’s changed by the time the fetch finishes.

property store: ObjectStore

Needed by dedup.pool_descriptor.PoolDescriptor.from_pool to describe an equivalent Pool for a multiprocess worker to rebuild — the same reason dedup.repository.DedupRepo already exposes its own store.

property pool_root: str

property vault_key: bytes | None

property verify_fingerprint: bool

This Pool’s own session-wide default — needed alongside store/pool_root/vault_key by dedup.pool_descriptor.PoolDescriptor.from_pool so a multiprocess worker’s own rebuilt Pool mirrors this one’s actual configuration instead of silently resetting it.

property verify_ciphertext_crc: bool

async bucket_path(stream_id, bucket_id)

Resolve (stream_id, bucket_id) to its physical .buk path (including any .<seqId> generation suffix), relative to the repository root.

async bucket(stream_id, bucket_id)

Return the (cached) BucketReader for (stream_id, bucket_id), opening and caching it on first access.

async open_bucket_uncached(stream_id, bucket_id)

Open (stream_id, bucket_id) fresh, without touching _buckets at all — no lookup, no insert, no eviction.

Exists for callers that need their own private, separately-scoped BucketReader cache instead of this shared one (a bulk sweep like export_scheduler.export_to’s bucket-major path, or verify’s Bucket-and-key stage — so a sweep touching every bucket once doesn’t evict this Pool’s own genuinely-hot interactive entries). _buckets itself is bound to call this for its own miss path (see Pool.__init__); the two never diverge in how a bucket actually gets opened, only in whether the result is remembered in the shared cache afterward.

async open_bucket_uncached_by_key(key)

open_bucket_uncached, taking its (stream_id, bucket_id) as one tuple — the single-arg shape AsyncKeyedCache.resolve’s fetch callback needs, so every caller building a BucketReader cache keyed this way (this class’s own _buckets, chunk_walk.py’s bucket-major export, verify’s Bucket stage) can pass this bound method directly instead of each writing its own lambda key: pool.open_bucket_uncached(*key).

async read_chunk(addr, *, cache=True, verify_fingerprint=None, verify_ciphertext_crc=None)

Resolve addr — using its own embedded stream_id/ bucket_id, not any caller-assumed values — to 4096 bytes of plaintext. Not used by the bucket-major export scheduler (chunk_walk.py’s _exec_one_bucket_group), which calls BucketReader.read_chunks directly, bypassing this class entirely.

cache=False bypasses the plaintext chunk cache entirely — for a one-off integrity check of a single chunk per bucket, which has nothing to gain from caching it and would only evict genuinely-hot interactive data for no benefit.

verify_fingerprint (None: defer to whatever this Pool was constructed with; an explicit True/False overrides it for this one call) compares this chunk’s plaintext SHA-256 against its stored .fgp fingerprint and raises DataCorruptError on a mismatch — a general data-integrity check independent of encryption, off by default since it costs one extra .inf/.fgp read per chunk. A plaintext-cache hit is checked too, not skipped, since the point is verifying the bytes about to be handed back.

verify_ciphertext_crc — forwarded to BucketReader.read_chunk as-is (None defers to whatever that reader was opened with). Checked before decrypting, so it still requires a vault key for an encrypted bucket to get as far as returning plaintext — a caller that wants the ChunkCrcStore check in isolation, independent of whether a key is even available, wants BucketReader.verify_chunk_ciphertext_crc instead.

cached_chunk(addr)

addr’s already-decoded plaintext if it’s currently in _chunks, else None – never fetches, never blocks (backed by AsyncKeyedCache’s own read-only Mapping.get). For a caller like dedup_file.py’s multi-chunk batch path that wants to skip re-fetching a chunk another read already populated, without going through read_chunk()’s own single-key fetch shape.

backfill_chunk(addr, plain)

The write half of cached_chunk’s peek: remembers addr’s already-decoded plain bytes in _chunks, as if a read_chunk(addr) call had fetched them. Does not itself verify a fingerprint – the caller is responsible for that before backfilling.

async verify_fingerprints(stream_id, bucket_id, chunks, *, verify_fingerprint=None)

Check every entry in chunks (already-decoded plaintext, keyed by its own chunk_idx within (stream_id, bucket_id)) against its stored .fgp digest — the same policy/lookup read_chunk applies to its own single chunk, factored out here so BucketReader.read_chunks’s multi-chunk batch result (fetched by dedup_file.py’s _fill_data_extent and chunk_walk.py’s _exec_one_bucket_group, both of which call read_chunks directly and so bypass read_chunk entirely) can honor it too, instead of silently skipping verification whenever a read/export spans more than one distinct chunk.

verify_fingerprint — same meaning as read_chunk’s own parameter: None defers to this Pool’s session-wide default.

Raises:

DataCorruptError – Any chunk’s plaintext SHA-256 doesn’t match its stored fingerprint.

async fingerprints_for(stream_id, bucket_id, chunk_indices)

Batched dedup.fingerprint.fingerprints(), using this Pool’s own AllocationTableCache — verify_fingerprints’s own lookup half (resolves the stored digests only, no plaintext comparison of its own), factored out so its own .inf/.fgp resolution shares this Pool’s cache the same way every other lookup here does.