synology_apm_repo.sdk.dedup.export_scheduler module¶
Export execution: turns a DedupFile/ByteRangeView’s own extents
into bytes on an ExportWriter.
Groups DATA chunks by (stream_id, bucket_id) via chunk_walk and
fetches each bucket’s chunks with merged BucketReader.read_chunk_blocks
reads instead of one Pool.read_chunk per chunk, since dedup means
logical offset order and Pool address order rarely match. By default the
groups decode in this process, ExportTuning.max_concurrent_reads at a
time, which hides a remote store’s latency. With ExportTuning.workers
set, a store that can be rebuilt in a fresh process
(PoolDescriptor.from_pool) and a writer offering a worker_target,
each DATA group’s decode+write moves to a ProcessPoolExecutor
(_WorkerPoolChoice decides per window; the worker side is
export_workers); gap/zero-fill writes always go through the writer
in-process.
- synology_apm_repo.sdk.dedup.export_scheduler.MAX_EXPORT_WORKERS = 61¶
what
ProcessPoolExecutorallows on Windows, so a count valid on one platform is valid on all.- Type:
The most
ExportTuning.workersan export accepts
- class synology_apm_repo.sdk.dedup.export_scheduler.ExportTuning(workers=0, max_concurrent_reads=4)¶
Bases:
objectHow an export of dedup-backed content decodes; the defaults suit most exports.
- Variables:
workers (int) – Worker processes decoding the content, at most
MAX_EXPORT_WORKERS;0decodes in this process. Workers pay off when one process can’t keep up: a destination faster than one process decodes, or one that each worker writes through its ownworker_targethandle. Each holds its own decode buffers, so memory grows with the count, and each export starts its own pool, so a folder export pays the start-up once per file. Workers are spawned and import the caller’s__main__: a script starting such an export does so only underif __name__ == "__main__":.max_concurrent_reads (int) – Bucket groups read and decoded at once in this process (
1: one at a time); several hide a remote store’s latency. A worker reads its own groups one at a time.
- Raises:
ValueError –
workersis outside0..MAX_EXPORT_WORKERSormax_concurrent_readsis less than 1.
Bases:
ExportTuningExportTuningplus the planning knobs and the state the calls of one logical export share (its segments, a disk’s fragments). The SDK’s own callers pass one throughContentSource.export_range, which forwards it like anyExportTuning;ofturns any tuning into one.- Variables:
window_entries (int) – Chunks per planning window (see
plan_chunks_windowed).max_concurrent_opens (int | None) – Prefetches upcoming bucket headers ahead of the main loop on the in-process path (
1disables it;None: off whenmax_concurrent_reads> 1, whose concurrent reads already overlap their opens, else 2). The prefetch runs through the whole window, so past the export cache’s size it evicts readers it opened and they are opened again.export_cache (synology_apm_repo.sdk.dedup.pool._cache.BucketReaderCache | None) – A private
BucketReaderCachefor in-process reads, kept separate fromPool’s shared one;offills it in, so every call handed the same instance keeps bucket-locality reuse across one logical export.executor (synology_apm_repo.sdk.dedup.export_workers.ExportExecutor | None) – The multiprocess workers.
Nonelets the call build its own whenworkersasks for one, and close it afterwards. A caller-supplied one serves every window, is left open, and must match both the file’s repository and the writer’s destination;workersis then ignored.
tuningresolved for an export: aSharedExportTuningas given, any other its settings (or the defaults), in both cases with anexport_cache, a new one when it carried none.
Whether an export with these settings builds its own worker pool (where its store and writer allow one):
workersasks for one and noexecutorwas given.
- async synology_apm_repo.sdk.dedup.export_scheduler.export_to_writer(file_like, writer, *, sparse=True, progress=None, tuning=None)¶
The export entry point for a
DedupFileorByteRangeView: writes the range intowriter, which the caller has opened at (at least) the range’s size and will commit or abort.DedupFile.export_range/ByteRangeView.export_rangedelegate here.A failure of a concurrent read or of a worker process is raised as itself (a further failure is noted on it), never as an
ExceptionGroup, and a worker that died asWorkerProcessError.- Parameters:
file_like (DedupContent) – The file or view to export.
writer (ExportWriter) – The destination, opened by the caller.
sparse (bool) – Leave
ZERO/HOLEranges unwritten. Ignored (zeros are written) for a writer whosecaps.supports_sparseisFalse; a dense export writes nothing for them either when the writer ispreallocated.progress (WrittenBytesCallback | None) – Async callback receiving each newly written count of real
DATAbytes, never holes/zeros.tuning (ExportTuning | None) – How the export decodes (default
ExportTuning()); aSharedExportTuningalso brings the state it shares with the other calls of one logical export.
- Returns:
The export’s byte totals.
- Raises:
ValueError – A
SharedExportTuning’sexecutorwas not built for this file’s repository and this writer’s destination.WorkerProcessError – A worker process died.
- Return type:
- async synology_apm_repo.sdk.dedup.export_scheduler.export_fragments_to_writer(fragments, writer, *, span=None, sparse=True, progress=None, tuning=None)¶
Exports each
(file, start, end)fragment, in order, intowriter(opened by the caller) throughexport_to_writer, each at its ownstartoffset.tuning’s concurrency applies within a fragment.The fragments share one
BucketReaderCache(aSharedExportTuning’sexport_cacheif given) and, when every fragment’s file resolves to the identicalPoolDescriptorand the writer takes worker writes, one multiprocess executor, built up front whatever the fragments’ size (aSharedExportTuning’sexecutorif given, left open; none whentuning.workersis0); otherwise each fragment’s export decides for itself.progressreceives each newly written count of realDATAbytes.- Parameters:
fragments (Sequence[tuple[DedupFile, int, int]]) – Disjoint
(file, start, end)pieces, each insidespan.writer (ExportWriter) – The destination, opened by the caller.
span (tuple[int, int] | None) – The
[start, end)range being exported. Offsets written towriterare relative to its start, and the gaps between fragments are holes of the result, zero-filled whereneeds_zero_fill.Noneexports the fragments alone, at their own offsets.
- Returns:
The fragments’ summed byte totals;
logical_sizeisspan’s length, or the fragments’ summed length without aspan.- Raises:
ValueError – A
SharedExportTuning’sexecutorwas not built for a fragment’s repository and this writer’s destination, or a fragment lies outsidespan.- Return type: