synology_apm_repo.sdk.dedup.buffered_export_sink module¶
BufferedExportSink: the base for a destination that accepts only linear
writes (a zip entry, an upload stream, the parts of a multipart upload).
The exporter writes in physical, not logical, order, so it cannot feed such a
destination directly. This sink takes each segment of the export in a buffer
it owns, where writes may land at any offset in any order; once the segment
is complete it is handed to flush_segment in the background, in segment
order, while the exporter fills the next one. At most max_buffered_segments
segments are held at once, so a slow destination holds the exporter back in
begin_segment instead of letting buffers pile up.
- class synology_apm_repo.sdk.dedup.buffered_export_sink.FlushableSegment(index, start, length, read)¶
Bases:
objectOne completed segment, handed to
BufferedExportSink.flush_segment.The segment is readable only until
flush_segmentreturns; anything kept past that must be a copy (read_allreturns one).- Variables:
index – The segment’s position in the export, counting from 0.
start – Where the segment starts in the export.
length – The segment’s size in bytes.
- async blocks(block=8388608)¶
Yields the segment’s bytes in order as views of at most
blockbytes, each released whenflush_segmentreturns.
- async read_all()¶
The whole segment, copied.
- class synology_apm_repo.sdk.dedup.buffered_export_sink.BufferedExportSink(segment_size, *, max_buffered_segments=2, storage='memory', spool_dir=None)¶
Bases:
ExportSinkAn
ExportSinkover a destination that takes bytes in order only.A subclass implements four hooks; this class owns the lifecycle, the buffers, the pacing and the background flushing. A failed flush fails the export, surfacing from the next
begin_segment, write,completeorcommit.abortwaits for the flush in progress, which cannot be interrupted safely, so it can take as long as one segment’s flush; only then isdiscard_destinationcalled.storagepicks where a segment’s bytes are held."memory"keeps them in a block of shared memory, which worker processes (an export withExportTuning.workers) write into directly as this process does."spool"keeps them in a temporary file, likewise writable by workers; it holds the same number of bytes on disk instead of in memory (the default temporary directory may itself be memory-backed: passspool_dirto choose one).- Parameters:
segment_size (int) – Bytes per segment, a multiple of 4096. Up to
segment_size * max_buffered_segmentsbytes are held at once.max_buffered_segments (int) – How many segments are held at once: the one being filled plus those waiting for or in the middle of their flush. At least 1; the default of 2 overlaps one flush with the next segment’s export.
storage (Literal['memory', 'spool']) –
"memory"or"spool".spool_dir (pathlib.Path | str | None) – The directory of the spool file;
Nonefor the system’s temporary directory. Only for"spool".
- Raises:
ValueError –
segment_sizeis not a positive multiple of 4096,max_buffered_segmentsis below 1,storageis unknown, orspool_diris given for memory storage.
- abstractmethod async create_destination(logical_size, *, sparse)¶
Creates the destination for an export of
logical_sizebytes. May raise to refuse it (a part-count limit, say);discard_destinationfollows, so it must cope with whatever a failed or cancelled call left.
- abstractmethod async flush_segment(segment)¶
Writes
segmentto the destination. Called once per segment, in segment order, never concurrently with itself.
- abstractmethod async finalize_destination()¶
Makes the destination final, after every segment was flushed.
- abstractmethod async discard_destination()¶
Releases the destination after a failure, once no flush is running.
- Returns:
Whether something of the export is left behind.
- Return type:
- property segment_size: int¶
Nonefor one segment spanning the whole export, else the bytes per segment, a multiple of 4096. Every segment but the last is exactly this long.
- async open(logical_size, *, sparse)¶
Creates the destination for an export of
logical_sizebytes.
- async begin_segment(index, start, length)¶
The writer for the segment covering
[start, start + length)of the export. Waits until the sink can take another segment, which is what paces the export. Indices count up from 0 and only one segment is open at a time.
- async commit()¶
Completes every handed-off write and makes the destination final.
- async abort()¶
Releases the destination after a failure. Safe to call in any state, including before
open, aftercommitand more than once; a repeated call returns the first call’s outcome.