synology_apm_repo.sdk.dedup.export_scheduler module

Export execution: turns a DedupFile/ByteRangeView’s own extents into bytes on an ExportWriter.

Groups DATA chunks by (stream_id, bucket_id) via chunk_walk and fetches each bucket’s chunks with merged BucketReader.read_chunk_blocks reads instead of one Pool.read_chunk per chunk, since dedup means logical offset order and Pool address order rarely match. By default the groups decode in this process, ExportTuning.max_concurrent_reads at a time, which hides a remote store’s latency. With ExportTuning.workers set, a store that can be rebuilt in a fresh process (PoolDescriptor.from_pool) and a writer offering a worker_target, each DATA group’s decode+write moves to a ProcessPoolExecutor (_WorkerPoolChoice decides per window; the worker side is export_workers); gap/zero-fill writes always go through the writer in-process.

synology_apm_repo.sdk.dedup.export_scheduler.MAX_EXPORT_WORKERS = 61

what ProcessPoolExecutor allows on Windows, so a count valid on one platform is valid on all.

Type:

The most ExportTuning.workers an export accepts

class synology_apm_repo.sdk.dedup.export_scheduler.ExportTuning(workers=0, max_concurrent_reads=4)

Bases: object

How an export of dedup-backed content decodes; the defaults suit most exports.

Variables:
  • workers (int) – Worker processes decoding the content, at most MAX_EXPORT_WORKERS; 0 decodes in this process. Workers pay off when one process can’t keep up: a destination faster than one process decodes, or one that each worker writes through its own worker_target handle. Each holds its own decode buffers, so memory grows with the count, and each export starts its own pool, so a folder export pays the start-up once per file. Workers are spawned and import the caller’s __main__: a script starting such an export does so only under if __name__ == "__main__":.

  • max_concurrent_reads (int) – Bucket groups read and decoded at once in this process (1: one at a time); several hide a remote store’s latency. A worker reads its own groups one at a time.

Raises:

ValueError – workers is outside 0..MAX_EXPORT_WORKERS or max_concurrent_reads is less than 1.

workers: int
max_concurrent_reads: int
class synology_apm_repo.sdk.dedup.export_scheduler.SharedExportTuning(workers=0, max_concurrent_reads=4, window_entries=1048576, max_concurrent_opens=None, export_cache=None, executor=None)

Bases: ExportTuning

ExportTuning plus the planning knobs and the state the calls of one logical export share (its segments, a disk’s fragments). The SDK’s own callers pass one through ContentSource.export_range, which forwards it like any ExportTuning; of turns any tuning into one.

Variables:
  • window_entries (int) – Chunks per planning window (see plan_chunks_windowed).

  • max_concurrent_opens (int | None) – Prefetches upcoming bucket headers ahead of the main loop on the in-process path (1 disables it; None: off when max_concurrent_reads > 1, whose concurrent reads already overlap their opens, else 2). The prefetch runs through the whole window, so past the export cache’s size it evicts readers it opened and they are opened again.

  • export_cache (synology_apm_repo.sdk.dedup.pool._cache.BucketReaderCache | None) – A private BucketReaderCache for in-process reads, kept separate from Pool’s shared one; of fills it in, so every call handed the same instance keeps bucket-locality reuse across one logical export.

  • executor (synology_apm_repo.sdk.dedup.export_workers.ExportExecutor | None) – The multiprocess workers. None lets the call build its own when workers asks for one, and close it afterwards. A caller-supplied one serves every window, is left open, and must match both the file’s repository and the writer’s destination; workers is then ignored.

window_entries: int
max_concurrent_opens: int | None
export_cache: BucketReaderCache | None
executor: ExportExecutor | None
classmethod of(tuning)

tuning resolved for an export: a SharedExportTuning as given, any other its settings (or the defaults), in both cases with an export_cache, a new one when it carried none.

property builds_workers: bool

Whether an export with these settings builds its own worker pool (where its store and writer allow one): workers asks for one and no executor was given.

async synology_apm_repo.sdk.dedup.export_scheduler.export_to_writer(file_like, writer, *, sparse=True, progress=None, tuning=None)

The export entry point for a DedupFile or ByteRangeView: writes the range into writer, which the caller has opened at (at least) the range’s size and will commit or abort. DedupFile.export_range/ByteRangeView.export_range delegate here.

A failure of a concurrent read or of a worker process is raised as itself (a further failure is noted on it), never as an ExceptionGroup, and a worker that died as WorkerProcessError.

Parameters:
  • file_like (DedupContent) – The file or view to export.

  • writer (ExportWriter) – The destination, opened by the caller.

  • sparse (bool) – Leave ZERO/HOLE ranges unwritten. Ignored (zeros are written) for a writer whose caps.supports_sparse is False; a dense export writes nothing for them either when the writer is preallocated.

  • progress (WrittenBytesCallback | None) – Async callback receiving each newly written count of real DATA bytes, never holes/zeros.

  • tuning (ExportTuning | None) – How the export decodes (default ExportTuning()); a SharedExportTuning also brings the state it shares with the other calls of one logical export.

Returns:

The export’s byte totals.

Raises:
  • ValueError – A SharedExportTuning’s executor was not built for this file’s repository and this writer’s destination.

  • WorkerProcessError – A worker process died.

Return type:

ExportResult

async synology_apm_repo.sdk.dedup.export_scheduler.export_fragments_to_writer(fragments, writer, *, span=None, sparse=True, progress=None, tuning=None)

Exports each (file, start, end) fragment, in order, into writer (opened by the caller) through export_to_writer, each at its own start offset. tuning’s concurrency applies within a fragment.

The fragments share one BucketReaderCache (a SharedExportTuning’s export_cache if given) and, when every fragment’s file resolves to the identical PoolDescriptor and the writer takes worker writes, one multiprocess executor, built up front whatever the fragments’ size (a SharedExportTuning’s executor if given, left open; none when tuning.workers is 0); otherwise each fragment’s export decides for itself. progress receives each newly written count of real DATA bytes.

Parameters:
  • fragments (Sequence[tuple[DedupFile, int, int]]) – Disjoint (file, start, end) pieces, each inside span.

  • writer (ExportWriter) – The destination, opened by the caller.

  • span (tuple[int, int] | None) – The [start, end) range being exported. Offsets written to writer are relative to its start, and the gaps between fragments are holes of the result, zero-filled where needs_zero_fill. None exports the fragments alone, at their own offsets.

Returns:

The fragments’ summed byte totals; logical_size is span’s length, or the fragments’ summed length without a span.

Raises:

ValueError – A SharedExportTuning’s executor was not built for a fragment’s repository and this writer’s destination, or a fragment lies outside span.

Return type:

ExportResult