synology_apm_repo.sdk.dedup.export_scheduler module

Export execution: turns a DedupFile/ByteRangeView’s own extents into bytes on a real destination file.

Dedup means “logical file offset ascending” is very often not “Pool address ascending” — an incremental VM backup’s later versions mostly INHERIT chunks scattered across whichever bucket happened to hold them at write time. Rather than walk extents in logical order and jump between hundreds of .buk files in essentially random order, this module always groups DATA chunks by (stream_id, bucket_id) via chunk_walk and fetches each bucket’s needed chunks with one merged BucketReader.read_chunks call instead of one Pool.read_chunk per chunk. A naive one-chunk-at-a-time path exists only as an independent correctness oracle in the test suite, never in production.

Every write goes through _ExportSink’s dedicated writer thread when running in a single process; see export_to for the max_concurrent_reads/max_concurrent_opens concurrency knobs. When this repository’s store can be reconstructed in a fresh process (dedup.pool_descriptor.PoolDescriptor.from_pool), every DATA chunk-group’s decode+write instead moves to a real ProcessPoolExecutor — real multi-core parallelism the single-process path never gets past CPython’s GIL for this CPU-bound work — leaving _ExportSink to handle only gap/zero-fill writes (cheap, not worth parallelizing) in-process either way. See ARCHITECTURE.md’s “Async-native, by design” section for the measurement this is based on — see _walk_bucket_major for a cost this windowed path specifically, and only it, accepts.

async synology_apm_repo.sdk.dedup.export_scheduler.export_to(file_like, dst, *, sparse=True, progress=None, window_entries=1048576, max_concurrent_opens=None, max_concurrent_reads=1, export_cache=None, dst_offset=0, create=True, executor=None)

The one export entry point for a whole DedupFile or a ByteRangeView window into one — grouping DATA chunks bucket-major via chunk_walk so each bucket’s needed chunks are fetched with one merged read instead of jumping between hundreds of .buk files in logical-offset order. DedupFile.export_to/ByteRangeView.export_to delegate here.

Progress reports real DATA bytes written against the range’s real DATA total (never holes/zeros), computed once via count_planned_bytes and skipped entirely when nobody’s listening.

max_concurrent_reads/max_concurrent_opens only govern the fallback path now (this repository’s store isn’t describable — see executor’s own paragraph below) — the real, unconditional multiprocess path’s cross-bucket concurrency is governed by concurrency.default_worker_count() instead. max_concurrent_reads (default 1: serial) is the single knob for every kind of read concurrency exec_chunks can produce on that fallback path, cross-bucket or in-bucket — see exec_chunks for the semaphore hand-off mechanics.

max_concurrent_opens (default None: auto-derived as 1 + max_concurrent_reads; an explicit integer, e.g. 1 to disable prefetch, overrides the derivation) prefetches upcoming buckets’ headers ahead of the main loop instead of overlapping full bucket-group executions on the fallback path — see _prefetch_bucket_opens for the mechanism. Whether raising it helps is backend-dependent (a saturated, low-RTT pipe gains nothing; a higher-RTT endpoint with spare bandwidth gains the most), but the auto-derived default stays on regardless, since raising it is never meaningfully slower even when it doesn’t help.

export_cache (default None: created here, bounded to DEFAULT_BUCKET_CACHE_SIZE and discarded at the end of this call) routes every fallback-path bucket-major read through a private BucketReaderCache instead of Pool’s own shared cache — see BucketReaderCache for why. VirtualDiskContentSource instead builds one instance (its own separately-bounded default) shared across all its fragments, so bucket-locality reuse persists across them on that path.

executor (default None: build one here, iff this repository’s store can be reconstructed in a fresh process, and tear it down before returning; given: the caller already built one and owns its lifetime — see concurrency.new_process_pool) is the real multiprocess dispatch this function uses unconditionally whenever it applies. Pass a shared one when calling this more than once for one logical export (VirtualDiskContentSource.export_to()’s own multi-fragment loop does exactly this) rather than let each call spin up its own pool independently.

dst_offset/create (default 0/True, the single-file behavior every other caller relies on) — dst_offset shifts every write’s destination position by a constant; its one real caller, pcps_disk, assembles several disk-absolute-addressed PC/PS fragments into one combined sparse disk image. create=False skips creating dst (the caller already created it at its own, larger, combined size, and calls this once per fragment into the same file — re-creating it here would destroy the previous fragment’s writes).

class synology_apm_repo.sdk.dedup.export_scheduler.ExportGroupWorkerArgs(stream_id, bucket_id, runs, size, dst_offset)

Bases: object

One (stream_id, bucket_id) group’s worth of work for export_bucket_group_worker — picklable (ChunkRun is already a plain dataclass of ints).

stream_id: StreamId

bucket_id: BucketId

runs: list[ChunkRun]

size: int

dst_offset: int

synology_apm_repo.sdk.dedup.export_scheduler.build_export_executor(descriptor, dst_path)

The one public factory for an executor this module’s own worker functions (above) know how to serve — for a caller (VirtualDiskContentSource.export_to()’s own multi-fragment fan-out) that wants to share one executor across several export_to() calls instead of letting each build (and tear down) its own. Kept here, not re-derived by that caller, so _export_worker_init itself stays module-private.

synology_apm_repo.sdk.dedup.export_scheduler.export_bucket_group_worker(args)

The multiprocess path’s per-bucket-group work item — returns the real number of bytes this group actually wrote (its caller’s own progress-reporting figure).