synology_apm_repo.sdk.dedup.pool package

Chunk pool: resolves a ChunkAddress to plaintext bytes.

The only place chunk decrypt+decompress happens. Three collaborating classes: BucketReader (_bucket_reader.py; one .buk file’s header/SizeStore/locators plus the per-chunk read), Pool (this module; the repository-wide entry point, which resolves (streamID, bucketID) to a .buk path and caches BucketReaders and decoded chunks), and BucketReaderCache (_cache.py; the private cache a bulk sweep uses instead of Pool’s own).

class synology_apm_repo.sdk.dedup.pool.BucketReader(store, path, header, index, vault_key, *, algorithm=None, verify_ciphertext_crc=False, sizestore_repaired=False, known_file_size=None)

Bases: object

One opened .buk file. Build with open.

The read paths use index’s raw columns directly, since a bucket-major export touches nearly every chunk.

Variables:
  • path – The .buk path in the store.

  • header – The parsed bucket header.

  • index – Every chunk’s SizeStore entry and byte range.

  • sizestore_repaired – open fixed a SizeStore CRC mismatch via Redundancy parity (check_bucket_structure reports it).

  • known_file_size – The on-disk size, when that repair had to fetch it.

async classmethod open(store, path, *, vault_key=None, algorithm=None, verify_ciphertext_crc=False)

Open path and parse its header + SizeStore in one read of up to COMPRESS_RESERVED_LENG (16384) bytes.

The SizeStore CRC is always verified. A mismatch first tries the bucket’s trailing Redundancy blob (_attempt_size_store_repair), so parity repair is transparent on every open; a successful repair sets sizestore_repaired.

Parameters:
  • store (ObjectStore) – The object store holding path.

  • path (str) – Physical .buk path.

  • vault_key (bytes | None) – Required to read chunks of a vault-encrypted bucket, even when a prebuilt AES algorithm is passed.

  • algorithm (AES | None) – A key schedule built from vault_key, reused by every chunk decrypt instead of rebuilding it.

  • verify_ciphertext_crc (bool) – Make every chunk read check its stored bytes against the ChunkCrcStore first; opening does not read the trailer itself.

Raises:
async ensure_chunk_crc_store()

Lazily read and self-validate this bucket’s ChunkCrcStore trailer (FORMAT-SPEC.md: ChunkCrcStore & Redundancy), caching the per-chunk ciphertext CRC32 values.

Returns:

The CRC32 values, one per stored chunk.

Raises:
  • FormatError – The trailer is shorter than declared.

  • DataCorruptError – The trailer’s bytes don’t match the header’s crcOfChunkCrc self-consistency field.

Return type:

tuple[int, …]

async verify_chunk_ciphertext_crc(chunk_idx)

Check chunk chunk_idx’s stored bytes against its ChunkCrcStore entry without decrypting, so no vault key is needed.

Raises:
async verify_raw_chunk_ciphertext_crc(chunk_idx, raw)

verify_chunk_ciphertext_crc for a caller that already holds chunk_idx’s stored bytes (e.g. from read_raw_chunks).

Raises:

DataCorruptError – raw doesn’t match chunk_idx’s own ChunkCrcStore entry.

non_compacted_chunk_ranges()

Every chunk with readable data, as ascending (chunk_idx_start, length) runs that skip COMPACTED slots (reclaimed, not corruption). Scans the raw compress-type bytes in C, not per chunk.

async read_chunk(chunk_idx, addr)

Decrypt (if vault-encrypted) and decompress chunk chunk_idx, first checking its stored bytes against its ChunkCrcStore entry (FORMAT-SPEC.md: ChunkCrcStore & Redundancy) when opened with verify_ciphertext_crc (one trailer read per bucket).

Parameters:
  • chunk_idx (ChunkIdx) – The chunk within this bucket.

  • addr (ChunkAddress) – The chunk’s own ChunkAddress, supplying the decryption IV.

Returns:

Exactly 4096 bytes of plaintext.

Raises:
Return type:

bytes

async decode_raw_chunk(chunk_idx, addr, raw)

read_chunk for a caller that already holds chunk_idx’s stored bytes (e.g. from read_raw_chunks).

async read_raw_chunk(chunk_idx)

Chunk chunk_idx’s stored bytes exactly as written (still compressed and/or encrypted), for ChunkCrcStore checks (FORMAT-SPEC.md: ChunkCrcStore & Redundancy).

Raises:
async read_raw_chunks(chunk_indices)

Batch form of read_raw_chunk: locator byte-ranges within _GAP_TOLERANCE of each other share one ObjectStore.read.

Parameters:

chunk_indices (Sequence[int]) – Sorted ascending and deduplicated.

Raises:
async read_chunks(stream_id, bucket_id, ranges)

Batch form of read_chunk: locator byte-ranges within _GAP_TOLERANCE of each other share one ObjectStore.read, the merged reads issued one after another; decrypt is per chunk and decompress is batched per merged run (_decode_chunks/decompress_many).

Parameters:
  • stream_id (StreamId) – This bucket’s stream, for an encrypted chunk’s IV.

  • bucket_id (BucketId) – This bucket’s id, likewise.

  • ranges (Sequence[tuple[int, int]]) – (chunk_idx_start, length) runs, sorted ascending and non-overlapping (not re-sorted here).

Returns:

Plaintext by chunk_idx. A value may be a memoryview into a buffer it shares with other chunks of its run; holding it keeps that buffer alive (copy with bytes() to keep one chunk).

Raises:
  • ChunkCompactedError – A requested chunk is COMPACTED; raised before any read.

  • KeyRequiredError – The bucket is encrypted and no vault key was given.

  • DataCorruptError – A range runs past this bucket’s chunks, a CRC check fails or a chunk doesn’t decompress to 4096 bytes.

Return type:

dict[int, bytes | memoryview]

async read_chunk_blocks(stream_id, bucket_id, ranges, on_block, *, semaphore=None)

read_chunks’ merged reads, each read’s chunks decoded and handed to on_block one DECODE_BLOCK_CHUNKS-aligned block at a time and dropped once it returns. A block’s chunks can arrive in several calls, one per merged read covering them; calls for different reads may run concurrently.

Parameters:
  • stream_id (StreamId) – This bucket’s stream, for an encrypted chunk’s IV.

  • bucket_id (BucketId) – This bucket’s id, likewise.

  • ranges (Sequence[tuple[int, int]]) – As for read_chunks.

  • on_block (Callable[[list[tuple[int, int]], dict[int, bytes | memoryview]], Awaitable[None]]) – Receives each block; a chunk may be a view into its read’s buffer, as read_chunks describes.

  • semaphore (Semaphore | None) – Shared read-concurrency bound (None: serial), the one exec_chunks() draws from. The caller has already acquired one permit for this call, which reads its merged runs in turn; a further run is read alongside only on a permit that is free when the call starts, never one it waits for. A permit is held while its read’s blocks are decoded and handed over.

Raises:
  • ChunkCompactedError – A requested chunk is COMPACTED; raised before any read.

  • KeyRequiredError – The bucket is encrypted and no vault key was given.

  • DataCorruptError – A range runs past this bucket’s chunks, a CRC check fails or a chunk doesn’t decompress to 4096 bytes.

class synology_apm_repo.sdk.dedup.pool.BucketReaderCache(maxsize=16)

Bases: AsyncKeyedCache[tuple[StreamId, BucketId], BucketReader]

Bucket-major export uses one instead of Pool’s bounded cache, so a sweep touching every bucket once doesn’t evict hot interactive entries. Read through Pool.bucket(..., cache=); never shared with Pool’s own cache. One instance spans every fragment of a VirtualDiskContentSource export; a standalone export_range gets its own.

Decoded chunk plaintext is not cached here: it stays scoped to one bucket-group call, which lets decompress_many return zero-copy memoryview values.

class synology_apm_repo.sdk.dedup.pool.Pool(store, pool_root, dir_cache, *, vault_key=None, limits=CacheLimits(bucket_readers=16, bucket_readers_interactive=64, chunks=4096, composition_records=64, composition_pages=128, saas_streams=8, dir_scan=256, dir_fixed=64, allocation_tables=256, forward_resolutions=4096), verify=VerifyPolicy(ciphertext_crc=False, fingerprint=False))

Bases: object

Repository-wide chunk pool entry point.

Resolves any ChunkAddress to its 4096-byte plaintext, with two-level LRU caching: BucketReaders (cheap, kept many) and decoded plaintext chunks (kept fewer). Both are shared AsyncKeyedCache instances, so concurrent callers missing the same key share one fetch; backfill_chunks is the one direct insert that bypasses that, so its callers take batch_fetch instead.

Parameters:
  • store (ObjectStore) – The repository’s ObjectStore.

  • pool_root (str) – Path of the @data/Pool directory.

  • dir_cache (DirCache) – Shared DirCache for .<seqId> resolution.

  • vault_key (bytes | None) – The VaultKey for an encrypted repository, else None.

  • limits (CacheLimits) – Bounds of the bucket-reader (bucket_readers), plaintext chunk (chunks) and allocation-table (allocation_tables) caches.

  • verify (VerifyPolicy) – The per-chunk checks every read runs.

release_caches()

Drop everything this pool holds in memory: decoded chunks, open bucket readers, and allocation tables. DirCache is untouched; it belongs to the store.

cache_stats()

Counters of this pool’s three caches, keyed pool.buckets, pool.chunks and pool.allocation_tables.

property release_epoch: int

Bumped once per release_caches() call. A caller fetching plaintext outside read_chunk() (dedup_file.py’s _resolve_bucket_group) captures it first and skips backfill_chunks if it changed.

property store: ObjectStore

The ObjectStore this pool reads from (used by PoolDescriptor.from_pool).

property pool_root: str
property vault_key: bytes | None
property limits: CacheLimits

The bounds this pool’s caches were built with.

property verify: VerifyPolicy

The per-chunk checks every read through this pool runs.

async bucket_path(stream_id, bucket_id)

Resolve (stream_id, bucket_id) to its store-relative physical .buk path, including any .<seqId> generation suffix.

Raises:

NotFoundError – No such bucket file.

async bucket_size(stream_id, bucket_id)

The on-disk size of bucket_path(stream_id, bucket_id) as its directory listing reported it, with no size request; None if the backend lists no sizes (store.size is then the way to ask). The listing is cached, so a store that changed after it was read needs Repository.invalidate_caches().

Raises:

NotFoundError – No such bucket file.

async bucket(stream_id, bucket_id, *, cache=None)

The BucketReader for (stream_id, bucket_id), opened on first access and cached in cache — a bulk sweep’s private one — or, when None, in this pool’s own.

async open_bucket_uncached(stream_id, bucket_id)

Open (stream_id, bucket_id) fresh, without touching this pool’s bucket cache (no lookup, insert or eviction).

Every bucket() cache miss, this pool’s own cache or a sweep’s BucketReaderCache, opens through it, as does a verify run’s bucket check.

async read_chunk(addr)

Resolve addr to 4096 bytes of plaintext, cached, with this pool’s verify checks (a cache hit’s fingerprint is checked too). Bucket-major export and verify FULL read through chunk_walk.read_verified_blocks instead.

Returns:

The plaintext chunk.

Raises:
Return type:

bytes

cached_chunks(stream_id, bucket_id, chunk_indices)

(hits, misses) for chunk_indices of one bucket: the plaintext already cached, by index, and the indices that are not; never fetches. Each lookup counts and refreshes recency as read_chunk()’s would. For batch callers (dedup_file.py) that skip read_chunk().

batch_fetch(stream_id, bucket_id)

Held around a batch caller’s cached_chunks-fetch-backfill_chunks of one bucket, so concurrent batches of the same bucket run one at a time and a later one finds what an earlier one cached instead of fetching it again.

backfill_chunks(stream_id, bucket_id, plaintexts)

Cache decoded plaintexts (by chunk_idx) of one bucket as if read_chunk() had fetched them; a memoryview is copied, so the cache never pins the buffer it views. Verifies no fingerprint; that is the caller’s job.

async verify_fingerprints(stream_id, bucket_id, chunks)

Check every entry in chunks (decoded plaintext keyed by chunk_idx) against its stored .fgp digest when this pool’s verify.fingerprint is on: read_chunk’s policy, for callers of BucketReader.read_chunks.

Parameters:
Raises:
  • DataCorruptError – Any chunk’s plaintext SHA-256 doesn’t match its stored fingerprint, or a sidecar is corrupt.

  • NotFoundError – The group’s .inf or .fgp file is missing.

  • FormatError – The .inf or .fgp file is truncated.

async expected_fingerprints(stream_id, bucket_id, chunk_indices)

The stored .fgp digest of every chunk_idx in chunk_indices, for check_fingerprints; None when this pool’s verify.fingerprint is off. Consecutive indices share one read.

Raises:
check_fingerprints(stream_id, bucket_id, chunks, expected)

Check every entry in chunks against its digest in expected (from expected_fingerprints), with no I/O.

Raises:

DataCorruptError – A chunk’s plaintext SHA-256 doesn’t match.

class synology_apm_repo.sdk.dedup.pool.VerifyPolicy(ciphertext_crc=False, fingerprint=False)

Bases: object

The per-chunk checks every read through a Pool runs. Browsing and export run neither (NO_VERIFY); VerifyLevel.FULL runs both (FULL_VERIFY) — see ARCHITECTURE.md’s validation table.

ciphertext_crc: bool

Check each chunk’s stored bytes against its bucket’s ChunkCrcStore, before decrypting.

fingerprint: bool

Check each decoded chunk’s SHA-256 against its stored .fgp fingerprint: the bucket group’s .inf table is read once, then one .fgp read per run of consecutive chunks checked.