synology_apm_repo.sdk.dedup.buffered_export_sink module

BufferedExportSink: the base for a destination that accepts only linear writes (a zip entry, an upload stream, the parts of a multipart upload).

The exporter writes in physical, not logical, order, so it cannot feed such a destination directly. This sink takes each segment of the export in a buffer it owns, where writes may land at any offset in any order; once the segment is complete it is handed to flush_segment in the background, in segment order, while the exporter fills the next one. At most max_buffered_segments segments are held at once, so a slow destination holds the exporter back in begin_segment instead of letting buffers pile up.

class synology_apm_repo.sdk.dedup.buffered_export_sink.FlushableSegment(index, start, length, read)

Bases: object

One completed segment, handed to BufferedExportSink.flush_segment.

The segment is readable only until flush_segment returns; anything kept past that must be a copy (read_all returns one).

Variables:
  • index – The segment’s position in the export, counting from 0.

  • start – Where the segment starts in the export.

  • length – The segment’s size in bytes.

async blocks(block=8388608)

Yields the segment’s bytes in order as views of at most block bytes, each released when flush_segment returns.

async read_all()

The whole segment, copied.

class synology_apm_repo.sdk.dedup.buffered_export_sink.BufferedExportSink(segment_size, *, max_buffered_segments=2, storage='memory', spool_dir=None)

Bases: ExportSink

An ExportSink over a destination that takes bytes in order only.

A subclass implements four hooks; this class owns the lifecycle, the buffers, the pacing and the background flushing. A failed flush fails the export, surfacing from the next begin_segment, write, complete or commit.

abort waits for the flush in progress, which cannot be interrupted safely, so it can take as long as one segment’s flush; only then is discard_destination called.

storage picks where a segment’s bytes are held. "memory" keeps them in a block of shared memory, which worker processes (an export with ExportTuning.workers) write into directly as this process does. "spool" keeps them in a temporary file, likewise writable by workers; it holds the same number of bytes on disk instead of in memory (the default temporary directory may itself be memory-backed: pass spool_dir to choose one).

Parameters:
  • segment_size (int) – Bytes per segment, a multiple of 4096. Up to segment_size * max_buffered_segments bytes are held at once.

  • max_buffered_segments (int) – How many segments are held at once: the one being filled plus those waiting for or in the middle of their flush. At least 1; the default of 2 overlaps one flush with the next segment’s export.

  • storage (Literal['memory', 'spool']) – "memory" or "spool".

  • spool_dir (pathlib.Path | str | None) – The directory of the spool file; None for the system’s temporary directory. Only for "spool".

Raises:

ValueError – segment_size is not a positive multiple of 4096, max_buffered_segments is below 1, storage is unknown, or spool_dir is given for memory storage.

abstractmethod async create_destination(logical_size, *, sparse)

Creates the destination for an export of logical_size bytes. May raise to refuse it (a part-count limit, say); discard_destination follows, so it must cope with whatever a failed or cancelled call left.

abstractmethod async flush_segment(segment)

Writes segment to the destination. Called once per segment, in segment order, never concurrently with itself.

abstractmethod async finalize_destination()

Makes the destination final, after every segment was flushed.

abstractmethod async discard_destination()

Releases the destination after a failure, once no flush is running.

Returns:

Whether something of the export is left behind.

Return type:

bool

property segment_size: int

None for one segment spanning the whole export, else the bytes per segment, a multiple of 4096. Every segment but the last is exactly this long.

async open(logical_size, *, sparse)

Creates the destination for an export of logical_size bytes.

async begin_segment(index, start, length)

The writer for the segment covering [start, start + length) of the export. Waits until the sink can take another segment, which is what paces the export. Indices count up from 0 and only one segment is open at a time.

async commit()

Completes every handed-off write and makes the destination final.

async abort()

Releases the destination after a failure. Safe to call in any state, including before open, after commit and more than once; a repeated call returns the first call’s outcome.