synology_apm_repo.sdk.dedup.chunk_walk module

The shared chunk-map walking engine: bucket-major (physical-order) export, plus the run splitting and bucket decode DedupFile.read and verify reuse.

plan_chunks_windowed (Pass 1) groups each DATA chunk placement by the (stream_id, bucket_id) it physically lives in, as compact ChunkRuns; exec_chunks (Pass 2) visits those groups bucket-major and decodes each unique chunk once, writing through a caller-supplied on_run(dest_offset, data) callback.

This module is the one deliberate exception to DedupFile._extents() being private (every call carries # noqa: SLF001): bucket-major planning needs the chunk-native addr/map_num/repeat fields DedupFile.read never exposes.

class synology_apm_repo.sdk.dedup.chunk_walk.ChunkRun(chunk_idx_start, length, dest_offset_start)

Bases: object

A maximal run of physically-contiguous chunks within one bucket: chunk_idx_start, chunk_idx_start + 1, …, mapping to dest_offset_start, dest_offset_start + FIXED_CHUNK_LENGTH, … in destination order.

Each repetition of a ChunkMapKind.MAPPING record’s address template (FORMAT-SPEC.md: ChunkMapRecord) is one run, split again where it crosses a bucket boundary (iter_chunk_runs).

chunk_idx_start: int
length: int
dest_offset_start: int
synology_apm_repo.sdk.dedup.chunk_walk.iter_chunk_runs(addr, map_num, first_k, last_k)

Yield (start_addr, run_length, k_of_start) for each maximal contiguous run within k in [first_k, last_k], splitting only where the repeat cycle wraps back to the template start (k % map_num == 0; FORMAT-SPEC.md: ChunkMapRecord) or ChunkAddress.advance carries into the next bucket_id.

synology_apm_repo.sdk.dedup.chunk_walk.data_extent_k_bounds(extent, start, end)

(first_k, last_k), the inclusive chunk-index bounds of a DATA extent clipped to [start, end): a chunk overlapping the range is included in full, one entirely outside is skipped.

class synology_apm_repo.sdk.dedup.chunk_walk.ChunkPlan(groups, holes, zeros)

Bases: object

Pass 1’s output: every DATA chunk placement in the range, grouped by the (stream_id, bucket_id) it lives in — ready for exec_chunks to visit bucket-major. holes/zeros are byte totals only; their zero-fill (if wanted) is already written by plan_chunks_windowed during the same walk.

groups: dict[tuple[StreamId, BucketId], list[ChunkRun]]
holes: int
zeros: int
synology_apm_repo.sdk.dedup.chunk_walk.DEFAULT_WINDOW_ENTRIES = 1048576

Default max_entries for plan_chunks_windowed, in chunks (not runs) per window. It bounds a plan’s memory however large the range.

async synology_apm_repo.sdk.dedup.chunk_walk.plan_chunks_windowed(base, start, end, window_start, *, write_zero_fill, max_entries=1048576)

One extent walk: DATA chunks are grouped as ChunkRuns for exec_chunks; ZERO/HOLE regions are resolved immediately by calling write_zero_fill(local_offset, length) (the caller decides whether that writes zeros or does nothing), or just counted when it is None.

Yields a ChunkPlan every max_entries chunks (a run counts its own length), splitting a run that crosses the boundary and carrying the remainder into the next window. Always yields at least one plan, possibly empty (a range that is entirely HOLE/ZERO).

ZERO/HOLE extents are clipped to [start, end); a boundary DATA extent is not, because chunks are the atomic decode unit: any chunk overlapping [start, end) is decoded in full.

Raises:

ValueError – window_start is not a multiple of FIXED_CHUNK_LENGTH, or is past start.

async synology_apm_repo.sdk.dedup.chunk_walk.iter_bucket_keys(base, start, end)

Every (stream_id, bucket_id) a DATA chunk of [start, end) lives in, in walk order, without building a plan; a key may repeat across extents. An extent spanning a whole repeat cycle is resolved from its map_num-chunk template once rather than per repetition.

async synology_apm_repo.sdk.dedup.chunk_walk.count_planned_bytes(base, start, end)

The DATA bytes plan_chunks_windowed would plan across [start, end): the repeat-expanded chunk count times FIXED_CHUNK_LENGTH.

An O(1)-memory walk of its own, so a progress denominator covers the whole range rather than one planning window.

synology_apm_repo.sdk.dedup.chunk_walk.merge_overlapping_ranges(ranges)

Collapse (chunk_idx_start, length) ranges that overlap or touch into their union, sorted ascending by start.

Two ChunkRuns of one bucket can reference overlapping physical ranges (dedup). The result is the sorted, duplicate-free request BucketReader.read_chunks/read_chunk_blocks require.

synology_apm_repo.sdk.dedup.chunk_walk.ascending_runs(chunk_indices)

Sorted, distinct chunk_indices as (start, length) runs of consecutive indices: merge_overlapping_ranges’ output shape, built in one linear pass from indices instead of from ranges.

async synology_apm_repo.sdk.dedup.chunk_walk.decode_bucket_chunks(reader, stream_id, bucket_id, ranges, *, pool)

Every chunk in ranges ((chunk_idx_start, length), overlap allowed) of reader’s bucket, decoded once each with merged reads and pool’s fingerprint policy applied, keyed by chunk index: for a caller-bounded read (DedupFile.read) that keeps the chunks; a sweep uses read_verified_blocks.

async synology_apm_repo.sdk.dedup.chunk_walk.read_verified_blocks(reader, stream_id, bucket_id, ranges, on_block, *, pool, semaphore=None)

BucketReader.read_chunk_blocks over ranges (sorted, duplicate-free, as merge_overlapping_ranges returns), each block checked against pool’s fingerprint policy before on_block gets it. The stored digests are read once for the whole request, once the first block is decoded: per-block checks add no .fgp reads, and a range or decode failure surfaces before any fingerprint read.

async synology_apm_repo.sdk.dedup.chunk_walk.exec_one_bucket_group(stream_id, bucket_id, runs, *, pool, on_run, size, export_cache, semaphore=None)

One (stream_id, bucket_id) group’s Pass-2 job: its chunks read via read_verified_blocks, each block flushed via on_run before the next is decoded, so the group never holds all its plaintext at once.

runs is visited twice: in chunk_idx order for reads, in dest_offset order within each block for writes. export_cache holds the BucketReader so the sweep never evicts Pool’s own interactive entries.

async synology_apm_repo.sdk.dedup.chunk_walk.exec_chunks(plan, *, pool, on_run, size, export_cache=None, max_concurrent_opens=1, max_concurrent_reads=1)

Pass 2: visits buckets ascending, chunks ascending within each, decoding each unique chunk once (see exec_one_bucket_group).

Parameters:
  • plan (ChunkPlan) – Pass 1’s output from plan_chunks_windowed.

  • pool (Pool) – Source of the BucketReaders and fingerprint policy.

  • on_run (Callable[[int, bytes | memoryview], Awaitable[None]]) – Called as on_run(dest_offset, data) for each merged run, with dest_offset relative to the plan’s window_start.

  • size (int) – Destination size in bytes; a run is truncated at size.

  • max_concurrent_reads (int) – Bound on in-flight reads, cross-bucket and in-bucket combined (default 1: serial). on_run must tolerate concurrent, out-of-order calls at scattered offsets when this is > 1.

  • max_concurrent_opens (int) – Above 1, runs _prefetch_bucket_opens in the background to warm export_cache (default 1: off).

  • export_cache (BucketReaderCache | None) – None creates one for this call (sized DEFAULT_LIMITS.bucket_readers); pass one to share it across calls.

Raises:

Exception – A bucket group’s own failure, never wrapped in an ExceptionGroup; with max_concurrent_reads > 1 the first one, noting how many others failed.