synology_apm_repo.sdk.dedup.composition_reader module

Composition sub-file access and record decoding.

Two collaborating classes:

  • CompositionReader — resolves (stream_id, session_id) to the right sequence of composition sub-files (via composition_path plus sequence-id resolution), and reads any byte range of the session’s global offset space, transparently crossing 16 MiB sub-file boundaries.

  • CompositionRecord — one backup version’s RecordHead plus binary-search-capable access to its ChunkMapRecord array — see CompositionRecord.entries for why it never linear-scans.

class synology_apm_repo.sdk.dedup.composition_reader.CompositionReader(store, dir_cache, comp_root, stream_id, session_id)

Bases: object

Read access to one (stream_id, session_id)’s composition sub-files. Does not validate the subID=0 header automatically on every access (that would cost a read for a check that virtually never fails and whose only failure mode — an addressing-constant mismatch — would already surface as consistently wrong data); call verify_header explicitly when that specific, stricter, opt-in check is wanted.

async read_at(global_offset, n)

Read n bytes starting at the session-global global_offset, transparently crossing 16 MiB sub-file boundaries (FORMAT-SPEC.md: composition-splitting).

async verify_header()

Explicitly validate the subID=0 sub-file’s 64-byte header (major version, subFileSize constant match).

async record(head_off)

Open the composition record at global offset head_off (db/file_map.comp_offset) — reads only the 32-byte RecordHead, not the (potentially huge) chunk-map array that follows it.

class synology_apm_repo.sdk.dedup.composition_reader.CompositionRecord(head_off, record_head, reader, _pages=<factory>, _page_end_offsets=<factory>, _contiguous_scanned=0)

Bases: object

One backup version’s composition record: a RecordHead plus lazy, page-cached, binary-search-capable access to its ChunkMapRecord array.

Pages (_PAGE_SIZE entries each) are fetched with one merged read and cached in _pages as raw bytes, bounded to _DEFAULT_PAGE_CACHE_MAXSIZE pages (LRU beyond that) — see _page_end_offsets for how _locate stays free of that bound. A page’s ChunkMapEntry values are materialized lazily, one at a time, only where a caller actually consumes them.

A caller that already has this record’s own chunk-map array bytes in hand and already knows they’re correct (seed_pages_from_array) can pre-populate the whole cache from them directly, instead of only ever filling it cold, page by page, off disk.

head_off: int
record_head: RecordHead
reader: CompositionReader
property status: CompositionStatus
property map_num: int
property attr_leng: int
async entries(start=0, end=None)

Stream ChunkMapEntry values whose range overlaps [start, end) (file offsets within the described file, not record indices). Locates the starting entry via _locate (binary search over known page boundaries, page-sized reads for anything not yet cached), then walks forward sequentially — never a linear, one-entry-at-a-time scan.

start=0, end=None (the default) walks every entry, for a full sequential export/verify.

async seed_pages_from_array(array_raw)

Pre-populate every page of this record’s cache from array_raw — chunk-map array bytes a caller (verify_checks.check_map_and_attr_crc) already confirmed correct via Redundancy-blob parity repair — instead of leaving _get_page to cold-fetch (and so re-derive from the still-corrupted on-disk copy) as entries() walks forward. array_raw must be exactly map_num * CHUNK_MAP_RECORD_LENGTH bytes.

A page already resolved by an earlier entries()/_locate()/ extent() call — e.g. units/device_pcps.py’s own open_disk(), which calls extent() (cold-fetching page 0 and the last page straight off disk) while assembling a PC/PS disk’s fragments, before this repaired record is ever seeded — is force-refreshed here too, not left with its stale content (see the loop below for how).

Raises:

ValueError – array_raw’s length doesn’t match this record’s own map_num — e.g. a concurrent writer changed the record between two independent reads of it, so the repair this is seeding from was computed against a differently shaped record than the one being seeded here.

async extent()

(start, end): the real file-offset range this record actually covers — its first entry’s file_offset to its last entry’s end_offset. Cheap regardless of map_num (two page reads, first and last, never a full scan).

Needed by PC/PS’s per-region disk fragments (units/content/pcps_disk.py): each fragment’s registered file_meta.file_size is the whole disk’s total capacity, identical across every sibling fragment, never that one fragment’s own real length — asking its own composition record directly is the only way to learn a fragment’s actual coverage (FORMAT-SPEC.md: pcps-fragments has the full story). VM’s model doesn’t need this: one composition record already covers the whole disk.