synology_apm_repo.sdk.dedup.export_sink module

The destination side of an export: where decoded bytes land.

ExportSink is a whole destination with a lifecycle; an ExportWriter is the write surface export_scheduler writes through, so an export can target something other than a local file (a hypervisor disk API, say) without touching the read/decode side. An export is cut into segments: the sink hands out one SegmentWriter per segment and is told when each is complete. RandomAccessExportSink (e.g. local_file_sink.LocalFileSink) is the one-segment case.

Lifecycle belongs to the caller, never to the scheduler: run_sink_export does open → body → commit and, on any failure, abort.

class synology_apm_repo.sdk.dedup.export_sink.SinkCaps(supports_sparse=True)

Bases: object

What a writer can and cannot do.

Variables:

supports_sparse (bool) – Ranges never written read back as zero, so the scheduler may skip HOLE/ZERO regions when the caller asks for a sparse export.

supports_sparse: bool
class synology_apm_repo.sdk.dedup.export_sink.AbortOutcome(kept, ever_written)

Bases: object

What ExportSink.abort left behind.

Variables:
  • kept (bool) – A partial destination was left in place.

  • ever_written (bool) – At least one write reached the sink before the abort.

kept: bool
ever_written: bool
class synology_apm_repo.sdk.dedup.export_sink.WorkerWriter(*args, **kwargs)

Bases: Protocol

A write handle a worker process opens for itself.

write_at(offset, data)
close()
class synology_apm_repo.sdk.dedup.export_sink.SinkDescriptor(*args, **kwargs)

Bases: Protocol

A picklable, value-comparable recipe for opening a WorkerWriter in another process. Two descriptors are equal exactly when their writers write to the same destination.

open_writer()
type synology_apm_repo.sdk.dedup.export_sink.WrittenBytesCallback = Callable[[int], Awaitable[None]]

Receives each newly written byte count of an export as it is handed to its writer.

class synology_apm_repo.sdk.dedup.export_sink.WorkerTarget(descriptor, base_offset=0)

Bases: object

Where a worker process writes on a sink’s behalf.

Variables:
descriptor: SinkDescriptor
base_offset: int
class synology_apm_repo.sdk.dedup.export_sink.ExportWriter(*args, **kwargs)

Bases: Protocol

An offset-addressed write surface for exported bytes: one whole destination (RandomAccessExportSink) or one segment of it (SegmentWriter).

Offsets are relative to the range being exported. Writes may arrive at any offset and in any order, and several may be in flight at once (an export reads ExportTuning.max_concurrent_reads bucket groups concurrently), so a writer whose write_at/write_zero awaits between steps on shared state serializes them itself. A write on a writer that is not open raises rather than being dropped. Offsets and lengths are multiples of 4096, except a write that ends at the end of its exported range, which may be shorter; a sector-addressed sink only needs to check in open that logical_size is a multiple of its sector size. write_at and write_zero may return before the bytes are durable; commit returning without raising means every handed-off write completed. write_zero’s length is unbounded (a hole can be gigabytes), so write it in blocks rather than allocating it at once, and zero whatever range it is given.

property caps: SinkCaps
property preallocated: bool

Every byte of the destination is already reserved and reads as zero, so a dense export has no zero-fill left to write. Only meaningful after open.

async write_at(offset, data)
async write_zero(offset, length)
worker_target()

Where worker processes may write directly, or None when every write must go through this writer in the parent process.

note_worker_write()

Called before work is handed to worker processes writing through worker_target, whose writes never reach write_at/write_zero: the destination may now hold data, which abort must report and keep accordingly.

class synology_apm_repo.sdk.dedup.export_sink.SegmentWriter(*args, **kwargs)

Bases: ExportWriter, Protocol

The writer for one segment, from ExportSink.begin_segment.

async complete()

Hands the segment’s bytes to the sink, which may flush them after this returns; every write to the segment has been made.

class synology_apm_repo.sdk.dedup.export_sink.ExportSink(*args, **kwargs)

Bases: Protocol

A destination for one export, fed segment by segment.

The caller owns the lifecycle (run_export): open creates the destination, each segment is written through the writer begin_segment returns and then complete``d, ``commit makes the destination final, abort follows a failure.

property segment_size: int | None

None for one segment spanning the whole export, else the bytes per segment, a multiple of 4096. Every segment but the last is exactly this long.

async open(logical_size, *, sparse)

Creates the destination for an export of logical_size bytes.

async begin_segment(index, start, length)

The writer for the segment covering [start, start + length) of the export. Waits until the sink can take another segment, which is what paces the export. Indices count up from 0 and only one segment is open at a time.

async commit()

Completes every handed-off write and makes the destination final.

async abort()

Releases the destination after a failure. Safe to call in any state, including before open, after commit and more than once; a repeated call returns the first call’s outcome.

class synology_apm_repo.sdk.dedup.export_sink.RandomAccessExportSink(*args, **kwargs)

Bases: SegmentWriter, ExportSink

Base for a destination that accepts writes at any offset in any order. Its one segment is the whole export and its writer is the sink itself, so a subclass implements open, commit, abort and the ExportWriter write surface. Every write stays in the parent process unless the subclass also overrides worker_target and note_worker_write.

segment_size: int | None = None
async begin_segment(index, start, length)

The writer for the segment covering [start, start + length) of the export. Waits until the sink can take another segment, which is what paces the export. Indices count up from 0 and only one segment is open at a time.

async complete()

Hands the segment’s bytes to the sink, which may flush them after this returns; every write to the segment has been made.

worker_target()

Where worker processes may write directly, or None when every write must go through this writer in the parent process.

note_worker_write()

Called before work is handed to worker processes writing through worker_target, whose writes never reach write_at/write_zero: the destination may now hold data, which abort must report and keep accordingly.

synology_apm_repo.sdk.dedup.export_sink.needs_zero_fill(writer, *, sparse)

Whether an export into writer must write its HOLE/ZERO ranges as zeros. A sparse export leaves them unwritten where the writer reads unwritten ranges as zero; a dense one writes them, unless the writer already reserved the whole destination as zero (ExportWriter.preallocated), which leaves the same bytes behind.

class synology_apm_repo.sdk.dedup.export_sink.SinkLifecycle(*args, **kwargs)

Bases: Protocol

The open/commit/abort an export’s caller drives.

async open(logical_size, *, sparse)
async commit()
async abort()
async synology_apm_repo.sdk.dedup.export_sink.run_sink_export(sink, logical_size, *, sparse, body)

open → body → commit, calling abort and re-raising on any failure (cancellation included). An error from abort never masks the original failure.

Parameters:
  • sink (SinkLifecycle) – The destination.

  • logical_size (int) – Size passed to sink.open.

  • sparse (bool) – Passed to sink.open.

  • body (Callable[[], Awaitable]) – Writes into the open sink and returns its result.

class synology_apm_repo.sdk.dedup.export_sink.OffsetWriter(inner, base)

Bases: object

Adds base (never negative) to every offset written through it, so sources that each address their own range can share one writer.

property caps: SinkCaps
property preallocated: bool
async write_at(offset, data)
async write_zero(offset, length)
worker_target()
note_worker_write()