synology_apm_repo.sdk.dedup.export_sink module¶
The destination side of an export: where decoded bytes land.
ExportSink is a whole destination with a lifecycle; an ExportWriter
is the write surface export_scheduler writes through, so an export can
target something other than a local file (a hypervisor disk API, say)
without touching the read/decode side. An export is cut into segments: the
sink hands out one SegmentWriter per segment and is told when each is
complete. RandomAccessExportSink (e.g. local_file_sink.LocalFileSink)
is the one-segment case.
Lifecycle belongs to the caller, never to the scheduler: run_sink_export
does open → body → commit and, on any failure, abort.
- class synology_apm_repo.sdk.dedup.export_sink.SinkCaps(supports_sparse=True)¶
Bases:
objectWhat a writer can and cannot do.
- Variables:
supports_sparse (bool) – Ranges never written read back as zero, so the scheduler may skip
HOLE/ZEROregions when the caller asks for a sparse export.
- class synology_apm_repo.sdk.dedup.export_sink.AbortOutcome(kept, ever_written)¶
Bases:
objectWhat
ExportSink.abortleft behind.- Variables:
- class synology_apm_repo.sdk.dedup.export_sink.WorkerWriter(*args, **kwargs)¶
Bases:
ProtocolA write handle a worker process opens for itself.
- write_at(offset, data)¶
- close()¶
- class synology_apm_repo.sdk.dedup.export_sink.SinkDescriptor(*args, **kwargs)¶
Bases:
ProtocolA picklable, value-comparable recipe for opening a
WorkerWriterin another process. Two descriptors are equal exactly when their writers write to the same destination.- open_writer()¶
- type synology_apm_repo.sdk.dedup.export_sink.WrittenBytesCallback = Callable[[int], Awaitable[None]]¶
Receives each newly written byte count of an export as it is handed to its writer.
- class synology_apm_repo.sdk.dedup.export_sink.WorkerTarget(descriptor, base_offset=0)¶
Bases:
objectWhere a worker process writes on a sink’s behalf.
- Variables:
descriptor (synology_apm_repo.sdk.dedup.export_sink.SinkDescriptor) – How the worker opens its own writer.
base_offset (int) – Added to every offset the worker writes at.
- descriptor: SinkDescriptor¶
- class synology_apm_repo.sdk.dedup.export_sink.ExportWriter(*args, **kwargs)¶
Bases:
ProtocolAn offset-addressed write surface for exported bytes: one whole destination (
RandomAccessExportSink) or one segment of it (SegmentWriter).Offsets are relative to the range being exported. Writes may arrive at any offset and in any order, and several may be in flight at once (an export reads
ExportTuning.max_concurrent_readsbucket groups concurrently), so a writer whosewrite_at/write_zeroawaits between steps on shared state serializes them itself. A write on a writer that is not open raises rather than being dropped. Offsets and lengths are multiples of 4096, except a write that ends at the end of its exported range, which may be shorter; a sector-addressed sink only needs to check inopenthatlogical_sizeis a multiple of its sector size.write_atandwrite_zeromay return before the bytes are durable;commitreturning without raising means every handed-off write completed.write_zero’slengthis unbounded (a hole can be gigabytes), so write it in blocks rather than allocating it at once, and zero whatever range it is given.- property preallocated: bool¶
Every byte of the destination is already reserved and reads as zero, so a dense export has no zero-fill left to write. Only meaningful after
open.
- async write_at(offset, data)¶
- async write_zero(offset, length)¶
- worker_target()¶
Where worker processes may write directly, or
Nonewhen every write must go through this writer in the parent process.
- note_worker_write()¶
Called before work is handed to worker processes writing through
worker_target, whose writes never reachwrite_at/write_zero: the destination may now hold data, whichabortmust report and keep accordingly.
- class synology_apm_repo.sdk.dedup.export_sink.SegmentWriter(*args, **kwargs)¶
Bases:
ExportWriter,ProtocolThe writer for one segment, from
ExportSink.begin_segment.- async complete()¶
Hands the segment’s bytes to the sink, which may flush them after this returns; every write to the segment has been made.
- class synology_apm_repo.sdk.dedup.export_sink.ExportSink(*args, **kwargs)¶
Bases:
ProtocolA destination for one export, fed segment by segment.
The caller owns the lifecycle (
run_export):opencreates the destination, each segment is written through the writerbegin_segmentreturns and thencomplete``d, ``commitmakes the destination final,abortfollows a failure.- property segment_size: int | None¶
Nonefor one segment spanning the whole export, else the bytes per segment, a multiple of 4096. Every segment but the last is exactly this long.
- async open(logical_size, *, sparse)¶
Creates the destination for an export of
logical_sizebytes.
- async begin_segment(index, start, length)¶
The writer for the segment covering
[start, start + length)of the export. Waits until the sink can take another segment, which is what paces the export. Indices count up from 0 and only one segment is open at a time.
- async commit()¶
Completes every handed-off write and makes the destination final.
- async abort()¶
Releases the destination after a failure. Safe to call in any state, including before
open, aftercommitand more than once; a repeated call returns the first call’s outcome.
- class synology_apm_repo.sdk.dedup.export_sink.RandomAccessExportSink(*args, **kwargs)¶
Bases:
SegmentWriter,ExportSinkBase for a destination that accepts writes at any offset in any order. Its one segment is the whole export and its writer is the sink itself, so a subclass implements
open,commit,abortand theExportWriterwrite surface. Every write stays in the parent process unless the subclass also overridesworker_targetandnote_worker_write.- async begin_segment(index, start, length)¶
The writer for the segment covering
[start, start + length)of the export. Waits until the sink can take another segment, which is what paces the export. Indices count up from 0 and only one segment is open at a time.
- async complete()¶
Hands the segment’s bytes to the sink, which may flush them after this returns; every write to the segment has been made.
- worker_target()¶
Where worker processes may write directly, or
Nonewhen every write must go through this writer in the parent process.
- note_worker_write()¶
Called before work is handed to worker processes writing through
worker_target, whose writes never reachwrite_at/write_zero: the destination may now hold data, whichabortmust report and keep accordingly.
- synology_apm_repo.sdk.dedup.export_sink.needs_zero_fill(writer, *, sparse)¶
Whether an export into
writermust write itsHOLE/ZEROranges as zeros. A sparse export leaves them unwritten where the writer reads unwritten ranges as zero; a dense one writes them, unless the writer already reserved the whole destination as zero (ExportWriter.preallocated), which leaves the same bytes behind.
- class synology_apm_repo.sdk.dedup.export_sink.SinkLifecycle(*args, **kwargs)¶
Bases:
ProtocolThe
open/commit/abortan export’s caller drives.- async open(logical_size, *, sparse)¶
- async commit()¶
- async abort()¶
- async synology_apm_repo.sdk.dedup.export_sink.run_sink_export(sink, logical_size, *, sparse, body)¶
open→body→commit, callingabortand re-raising on any failure (cancellation included). An error fromabortnever masks the original failure.- Parameters:
sink (SinkLifecycle) – The destination.
logical_size (int) – Size passed to
sink.open.sparse (bool) – Passed to
sink.open.body (Callable[[], Awaitable]) – Writes into the open sink and returns its result.
- class synology_apm_repo.sdk.dedup.export_sink.OffsetWriter(inner, base)¶
Bases:
objectAdds
base(never negative) to every offset written through it, so sources that each address their own range can share one writer.- async write_at(offset, data)¶
- async write_zero(offset, length)¶
- worker_target()¶
- note_worker_write()¶