synology_apm_repo.sdk.dedup.dedup_file module

DedupFile: the sole cross-layer read contract.

Any workload — VM disk image, FS file, SaaS raw object — ultimately reduces to a (stream_id, session_id, comp_offset) triple plus a size. Once you have that, everything above this module (catalog, units, CLI, TUI) only ever calls read()/stream()/export_to() — never touches a bucket, chunk, or composition record directly. ByteRangeView is the second shared primitive: FS’s content_dedup_id + file_size and SaaS’s object_table.(offset, length) are the same concept (a named sub-range of a bigger dedup file), so they share this one class instead of each workload reinventing it.

Performance-critical property: every read()/extents() call resolves its starting chunk-map record via CompositionRecord.entries’s binary search — over already-known page boundaries, page-sized reads for anything not yet cached — never a linear, one-entry-at-a-time scan.

synology_apm_repo.sdk.dedup.dedup_file.DEFAULT_STREAM_BLOCK = 8388608

Shared by every real stream_via_read caller across this project’s own ContentSource implementers (DedupFile, ByteRangeView, and — one layer up, in units/ — VirtualDiskContentSource/DissectFileContentSource/ _LocalFileContentSource) as their own stream()’s default block size, so the literal is defined once instead of independently redeclared per module.

async synology_apm_repo.sdk.dedup.dedup_file.stream_via_read(source, block=8388608)

Yield (offset, bytes) blocks front-to-back via repeated source.read(offset, block) calls — just a loop, so any implementer whose own read() already handles its internal chunking/extents/fragments (DedupFile’s own bucket-merged reads via _fill_data_extent, VirtualDiskContentSource’s fragment stitching, …) gets a correct stream() for free, with no separate bulk-read path to keep in sync. ContentSource is deliberately a structural Protocol, not an ABC, so implementers don’t need a common base class — but nothing then forced them to share this identical loop either; this closes that gap without adding one. Not used by every ContentSource implementer: LazyArtifact (assembled .eml/.ics content) has its own, different stream() that slices an already-materialized in-memory buffer instead, since its own size is None until that buffer is built.

Raises ValueError if source.size is None — every real caller here always has a known size by the time streaming starts.

synology_apm_repo.sdk.dedup.dedup_file.clamp_read_length(offset, length, size)

Resolve a read(offset, length) request against a known size: fills in the default length (offset to size) and clamps a request extending past size down to what’s actually there. Every ContentSource implementer shares this contract: reading past the end returns fewer bytes, never an error — only a negative offset/length is left for the caller to reject before calling this.

class synology_apm_repo.sdk.dedup.dedup_file.ExtentKind(value)

Bases: Enum

A file, as seen through DedupFile.extents(), is a sequence of these three kinds. ZERO (an explicit ChunkMapKind.ZERO chunk-map record) and HOLE (no record at all, i.e. a gap between records) are deliberately distinct even though both read back as zero bytes — a HOLE is a real sparse gap worth preserving as one on export; a ZERO record is zero data explicitly recorded as such, not merely absent.

DATA = 1

ZERO = 2

HOLE = 3

class synology_apm_repo.sdk.dedup.dedup_file.Extent(offset, length, kind, addr=None, map_num=0, repeat=0)

Bases: object

One contiguous span of a DedupFile. addr/map_num/ repeat are populated only for ExtentKind.DATA — the template starting address and its 1 + repeat repetitions the span’s chunks are drawn from (ChunkAddress.advance’s carry semantics).

offset: int

length: int

kind: ExtentKind

addr: ChunkAddress | None = None

map_num: int = 0

repeat: int = 0

property end: int

class synology_apm_repo.sdk.dedup.dedup_file.ExportResult(bytes_written, logical_size, holes, zeros)

Bases: object

The outcome of one export_to() call.

Variables:

• bytes_written (int) – Actual bytes written to the destination.

• logical_size (int) – The exported file’s full logical size.

• holes (int) – Byte count skipped via a sparse hole instead of writing zeros.

• zeros (int) – Byte count written as explicit zero bytes (not sparse).

bytes_written: int

logical_size: int

holes: int

zeros: int

class synology_apm_repo.sdk.dedup.dedup_file.DedupFile(comp_reader, pool, comp_offset, *, size=None)

Bases: object

A logical, byte-addressable file backed by one composition record.

size is supplied by the caller (file_meta.file_size, object_table.file_size, …) — this layer never infers it from the composition record’s own coverage, since a record’s last byte is not necessarily the file’s declared end (trailing HOLE).

property stream_id: int

property session_id: int

property comp_offset: int

property pool: Pool

The Pool this file’s chunks resolve through — needed by export_scheduler.py’s bucket-major planning, which schedules against a shared Pool/ BucketReaderCache rather than one read at a time.

async cached_record()

This file’s own CompositionRecord, fetched once and cached for the life of this DedupFile — a cold fetch on the first call, the cached instance on every one after.

Exposed (not fully private) because CompositionRecord is a real, shared unit with this class rather than an implementation detail confined to it: units/device_pcps.py and units/verify_reachable.py both need direct access to it (the latter also via seed_record(), to share one already-resolved record across several DedupFile``s that address the same composition) — one documented accessor here instead of each reaching into ``._record on its own.

seed_record(record)

Sets this file’s own cached CompositionRecord directly, skipping the fetch cached_record() would otherwise make — for a caller (units/verify_reachable.py) that already resolved the same composition’s record via a different DedupFile sharing the same key, and wants every DedupFile addressing it to reuse that one instance rather than each re-fetching its own copy.

async read(offset=0, length=None)

Read length bytes starting at offset (default: from offset to size). ZERO/HOLE extents read back as zero bytes for free (bytearray’s own zero-fill). A request extending past size is clamped to the bytes that actually exist (see clamp_read_length), never zero-padded past it.

Raises:

ValueError – offset or length is negative.

stream(block=8388608)

Yield (offset, bytes) blocks front-to-back — just a read(offset, block) loop, since read() already handles its own chunking, so it inherits _fill_data_extent’s own bucket-merged reads for free; no separate bulk-read path to keep in sync.

async export_to(dst, *, sparse=True, progress=None, window_entries=None, max_concurrent_opens=None, max_concurrent_reads=1, export_cache=None, dst_offset=0, create=True, executor=None)

Write this file to dst. sparse=True (the default) never writes ZERO/HOLE bytes at all — the output file reads back as zero there via the filesystem’s own sparse-hole support, and the two kinds are indistinguishable to a reader either way; use sparse=False when the caller needs the on-disk allocation to actually match the logical size. See export_scheduler.export_to for what window_entries/max_concurrent_opens/ max_concurrent_reads/export_cache/dst_offset/ create/executor do.

No verify_map_crc option here (or anywhere in the export path) — chunk-map CRC validation is verify’s job specifically (synology-apm-repo-cli verify --level full), not export’s: FULL already does an equally thorough, equally costly check, so export re-running it would only double that cost for no new information.

view(offset, length)

property supports_concurrent_export: bool

Always True — a bucket-backed read has a real bucket concept to spread reads across (see ContentSource.supports_concurrent_export).

class synology_apm_repo.sdk.dedup.dedup_file.ByteRangeView(base, offset, length)

Bases: object

A named sub-range of a DedupFile — the same read-contract shape (size/read/stream/export_to), coordinates translated, for a workload whose “file” is really just an offset+length window into a bigger dedup file rather than its own composition record.

property base: DedupFile

The DedupFile this view windows into — needed by export_scheduler.py’s bucket-major planning, which schedules against the base file’s own DedupFile.pool.

property offset: int

This view’s start offset within base.

async read(offset=0, length=None)

See units.base.ContentSource.read’s EOF contract — a request extending past this view’s own size is clamped, never an error.

stream(block=8388608)

async export_to(dst, *, sparse=True, progress=None, window_entries=None, max_concurrent_opens=None, max_concurrent_reads=1, export_cache=None, dst_offset=0, create=True, executor=None)

Same as DedupFile.export_to, windowed to this view’s own range.

property supports_concurrent_export: bool

Always True — same bucket-backed shape as the base DedupFile it windows into.