synology_apm_repo.sdk.dedup.chunk_walk module¶
The shared chunk-map walking engine: bucket-major (physical-order)
export, plus the run splitting and bucket decode DedupFile.read and
verify reuse.
plan_chunks_windowed (Pass 1) groups each DATA chunk placement by
the (stream_id, bucket_id) it physically lives in, as compact
ChunkRuns; exec_chunks (Pass 2) visits those groups
bucket-major and decodes each unique chunk once, writing through a
caller-supplied on_run(dest_offset, data) callback.
This module is the one deliberate exception to DedupFile._extents()
being private (every call carries # noqa: SLF001): bucket-major
planning needs the chunk-native addr/map_num/repeat fields
DedupFile.read never exposes.
- class synology_apm_repo.sdk.dedup.chunk_walk.ChunkRun(chunk_idx_start, length, dest_offset_start)¶
Bases:
objectA maximal run of physically-contiguous chunks within one bucket:
chunk_idx_start,chunk_idx_start + 1, …, mapping todest_offset_start,dest_offset_start + FIXED_CHUNK_LENGTH, … in destination order.Each repetition of a
ChunkMapKind.MAPPINGrecord’s address template (FORMAT-SPEC.md: ChunkMapRecord) is one run, split again where it crosses a bucket boundary (iter_chunk_runs).
- synology_apm_repo.sdk.dedup.chunk_walk.iter_chunk_runs(addr, map_num, first_k, last_k)¶
Yield
(start_addr, run_length, k_of_start)for each maximal contiguous run withink in [first_k, last_k], splitting only where the repeat cycle wraps back to the template start (k % map_num == 0; FORMAT-SPEC.md: ChunkMapRecord) orChunkAddress.advancecarries into the nextbucket_id.
- synology_apm_repo.sdk.dedup.chunk_walk.data_extent_k_bounds(extent, start, end)¶
(first_k, last_k), the inclusive chunk-index bounds of aDATAextent clipped to[start, end): a chunk overlapping the range is included in full, one entirely outside is skipped.
- class synology_apm_repo.sdk.dedup.chunk_walk.ChunkPlan(groups, holes, zeros)¶
Bases:
objectPass 1’s output: every
DATAchunk placement in the range, grouped by the(stream_id, bucket_id)it lives in — ready forexec_chunksto visit bucket-major.holes/zerosare byte totals only; their zero-fill (if wanted) is already written byplan_chunks_windowedduring the same walk.
- synology_apm_repo.sdk.dedup.chunk_walk.DEFAULT_WINDOW_ENTRIES = 1048576¶
Default
max_entriesforplan_chunks_windowed, in chunks (not runs) per window. It bounds a plan’s memory however large the range.
- async synology_apm_repo.sdk.dedup.chunk_walk.plan_chunks_windowed(base, start, end, window_start, *, write_zero_fill, max_entries=1048576)¶
One extent walk:
DATAchunks are grouped asChunkRuns forexec_chunks;ZERO/HOLEregions are resolved immediately by callingwrite_zero_fill(local_offset, length)(the caller decides whether that writes zeros or does nothing), or just counted when it isNone.Yields a
ChunkPlaneverymax_entrieschunks (a run counts its ownlength), splitting a run that crosses the boundary and carrying the remainder into the next window. Always yields at least one plan, possibly empty (a range that is entirelyHOLE/ZERO).ZERO/HOLEextents are clipped to[start, end); a boundaryDATAextent is not, because chunks are the atomic decode unit: any chunk overlapping[start, end)is decoded in full.- Raises:
ValueError –
window_startis not a multiple ofFIXED_CHUNK_LENGTH, or is paststart.
- async synology_apm_repo.sdk.dedup.chunk_walk.iter_bucket_keys(base, start, end)¶
Every
(stream_id, bucket_id)aDATAchunk of[start, end)lives in, in walk order, without building a plan; a key may repeat across extents. An extent spanning a whole repeat cycle is resolved from itsmap_num-chunk template once rather than per repetition.
- async synology_apm_repo.sdk.dedup.chunk_walk.count_planned_bytes(base, start, end)¶
The
DATAbytesplan_chunks_windowedwould plan across[start, end): therepeat-expanded chunk count timesFIXED_CHUNK_LENGTH.An O(1)-memory walk of its own, so a progress denominator covers the whole range rather than one planning window.
- synology_apm_repo.sdk.dedup.chunk_walk.merge_overlapping_ranges(ranges)¶
Collapse
(chunk_idx_start, length)ranges that overlap or touch into their union, sorted ascending by start.Two
ChunkRuns of one bucket can reference overlapping physical ranges (dedup). The result is the sorted, duplicate-free requestBucketReader.read_chunks/read_chunk_blocksrequire.
- synology_apm_repo.sdk.dedup.chunk_walk.ascending_runs(chunk_indices)¶
Sorted, distinct
chunk_indicesas(start, length)runs of consecutive indices:merge_overlapping_ranges’ output shape, built in one linear pass from indices instead of from ranges.
- async synology_apm_repo.sdk.dedup.chunk_walk.decode_bucket_chunks(reader, stream_id, bucket_id, ranges, *, pool)¶
Every chunk in
ranges((chunk_idx_start, length), overlap allowed) ofreader’s bucket, decoded once each with merged reads andpool’s fingerprint policy applied, keyed by chunk index: for a caller-bounded read (DedupFile.read) that keeps the chunks; a sweep usesread_verified_blocks.
- async synology_apm_repo.sdk.dedup.chunk_walk.read_verified_blocks(reader, stream_id, bucket_id, ranges, on_block, *, pool, semaphore=None)¶
BucketReader.read_chunk_blocksoverranges(sorted, duplicate-free, asmerge_overlapping_rangesreturns), each block checked againstpool’s fingerprint policy beforeon_blockgets it. The stored digests are read once for the whole request, once the first block is decoded: per-block checks add no.fgpreads, and a range or decode failure surfaces before any fingerprint read.
- async synology_apm_repo.sdk.dedup.chunk_walk.exec_one_bucket_group(stream_id, bucket_id, runs, *, pool, on_run, size, export_cache, semaphore=None)¶
One
(stream_id, bucket_id)group’s Pass-2 job: its chunks read viaread_verified_blocks, each block flushed viaon_runbefore the next is decoded, so the group never holds all its plaintext at once.runsis visited twice: inchunk_idxorder for reads, indest_offsetorder within each block for writes.export_cacheholds theBucketReaderso the sweep never evictsPool’s own interactive entries.
- async synology_apm_repo.sdk.dedup.chunk_walk.exec_chunks(plan, *, pool, on_run, size, export_cache=None, max_concurrent_opens=1, max_concurrent_reads=1)¶
Pass 2: visits buckets ascending, chunks ascending within each, decoding each unique chunk once (see
exec_one_bucket_group).- Parameters:
plan (ChunkPlan) – Pass 1’s output from
plan_chunks_windowed.pool (Pool) – Source of the
BucketReaders and fingerprint policy.on_run (Callable[[int, bytes | memoryview], Awaitable[None]]) – Called as
on_run(dest_offset, data)for each merged run, withdest_offsetrelative to the plan’swindow_start.size (int) – Destination size in bytes; a run is truncated at
size.max_concurrent_reads (int) – Bound on in-flight reads, cross-bucket and in-bucket combined (default 1: serial).
on_runmust tolerate concurrent, out-of-order calls at scattered offsets when this is> 1.max_concurrent_opens (int) – Above 1, runs
_prefetch_bucket_opensin the background to warmexport_cache(default 1: off).export_cache (BucketReaderCache | None) –
Nonecreates one for this call (sizedDEFAULT_LIMITS.bucket_readers); pass one to share it across calls.
- Raises:
Exception – A bucket group’s own failure, never wrapped in an
ExceptionGroup; withmax_concurrent_reads > 1the first one, noting how many others failed.