synology_apm_repo.sdk.dedup.verify_checks module

Byte-level integrity check primitives for units/verify_reachable.py’s top-down, reachability-scoped walk — what Repository.verify()/ Catalog.verify() actually use.

Every function here is a pure “given this already-open reader/triple, check it, return the Finding``(s)" primitive — no scanning of ``db/file_map, no bucket sampling, no notion of which triples/buckets are worth looking at in the first place. That’s the walker’s own job; this module only owns how to check one thing once it’s been decided the thing is worth checking.

Also owns the Finding/Stage/Symptom/VerifyLevel value types the walker (and its callers, api.catalog/api.repository) share — defined here, at the bottom of the dependency, so nothing above this module ever needs to import back into units/verify_reachable.py just to reference one of these value types.

class synology_apm_repo.sdk.dedup.verify_checks.Symptom(value)

Bases: Enum

The symptom categories a finding can carry.

CORRUPTION = 'Corruption'

FILE_MISSING = 'FileMissing'

MISMATCH = 'Mismatch'

DATA_MISSING = 'DataMissing'

KEY_MISSING = 'KeyMissing'

REPAIRED_VIA_PARITY = 'RepairedViaParity'

A CRC mismatch that this SDK’s own Redundancy-blob self-repair (format.redundancy.attempt_repair) was able to reconstruct and byte-for-byte confirm correct. Not a problem left unresolved, but still reported (not silently dropped) since repeated repairs against the same bucket group over time is itself a signal of failing underlying storage, even though the check the caller actually cared about (is this data readable) passed.

class synology_apm_repo.sdk.dedup.verify_checks.VerifyLevel(value)

Bases: Enum

How thorough a verify run is. FULL reads and checks every live chunk in a touched bucket — both its ciphertext CRC32 and, once a vault key is available, its decrypt+decompress+SHA-256 fingerprint — as costly as a real full export of the whole repository. QUICK reads no chunk content at all, only each touched bucket’s own structural checks; there is deliberately no sampled middle tier.

QUICK = 'quick'

FULL = 'full'

class synology_apm_repo.sdk.dedup.verify_checks.Stage(value)

Bases: StrEnum

The stages a Finding can come from. A str subclass: every existing site rendering/serializing Finding.stage directly as a string keeps working unchanged.

REPO_INFO = 'RepoInfo'

FILE_MAP = 'FileMap'

COMPOSITION = 'Composition'

BUCKET = 'Bucket'

ENCRYPT_KEY = 'EncryptKey'

VERSION = 'Version'

A catalog/workload/version that should resolve to real content, per its own metadata, but doesn’t — covers failure at either the version level or the workload/connection-enumeration level above it, one stage rather than one per layer of a top-down walk.

class synology_apm_repo.sdk.dedup.verify_checks.Finding(stage, symptom, path, detail, ref=None)

Bases: object

One integrity-check result. path is whatever store-relative path (or file_map path, or workload/version display info, for a Stage.VERSION finding) the finding is about.

ref (None unless set) is a canonical repo_path#cat:<id>/wl:<id>/ver:<uid> ref (units.node_ref.NodeRef) naming the exact catalog/workload/version this finding was found while checking — set by units/verify_reachable.py’s top-down walk, the only place with a Version in hand to name. When a bucket/chunk is shared by more than one version through dedup, ref names whichever version’s own check actually triggered this finding first — memoization means a later version sharing the same already-checked data produces no finding of its own, not a second one with a different ref.

stage: Stage

symptom: Symptom

path: str

detail: str

ref: str | None = None

async synology_apm_repo.sdk.dedup.verify_checks.check_repo_info(repo)

repo_info’s own header/CRC/JSON-parse check — repository-wide, not scoped to any one catalog reference, so verify_reachable() runs this exactly once regardless of what else it covers.

async synology_apm_repo.sdk.dedup.verify_checks.check_composition_header(reader, *, path)

The session’s own subID=0 sub-file header (CompositionReader.verify_header) — the stricter, opt-in check CompositionReader itself normally skips. Meant to be called once per distinct (stream_id, session_id), not once per row/extent that happens to share it.

async synology_apm_repo.sdk.dedup.verify_checks.check_record_head(reader, comp_offset, *, path)

Read and parse the 32-byte RecordHead at comp_offset — magic + head_crc are checked unconditionally by parse_record_head itself. Returns (finding, None) on failure, (None, record_head) on success.

async synology_apm_repo.sdk.dedup.verify_checks.verify_chunk_map_crc_threaded(map_array_bytes, expected_crc)

format.composition.verify_chunk_map_crc, with the thread-hop decision folded in — map_array_bytes can be a multi-hundred-MB chunk-map array, so this keeps that rare case’s zlib.crc32 pass off the event loop the same way every other large-payload synchronous computation in this codebase does (catalog/version.py::open_target_db, units/fs.py’s own peel() call). should_thread_chunk_map_crc skips the hop for the common small-array case, where the hop itself would cost more than the crc32 pass it’s saving. Shared by this module’s own check_map_and_attr_crc and diagnostics.py’s _verify_map_crc — same check, two orchestrators.

Raises:

DataCorruptError – map_array_bytes doesn’t match expected_crc.

async synology_apm_repo.sdk.dedup.verify_checks.check_map_and_attr_crc(reader, comp_offset, record_head, *, path)

Full mapCrc over the whole chunk-map array plus (when present) attrCrc over the trailing JSON attribute blob — one merged read for both regions, since they’re contiguous.

A map_crc mismatch is not immediately reported as a problem: this first tries the record’s own Redundancy-blob self-repair (_attempt_map_crc_repair), since every current-format record carries one (RecordHead.has_redundancy, checked by parse_record_head itself). A confirmed-correct repair is reported as Symptom.REPAIRED_VIA_PARITY (see that symptom for why a successful repair still gets reported) rather than dropped silently.

Nothing is checked when record_head.map_num == 0 — an empty chunk-map array has nothing to CRC.

Returns:

(findings, repaired_array). repaired_array is the confirmed-correct chunk-map array bytes exactly when a map_crc mismatch was just repaired via parity, None otherwise (no mismatch, or one that couldn’t be repaired). A caller with its own separate re-read of this same array — units/verify_reachable.py’s plan_chunks_windowed walk goes through a different CompositionReader/CompositionRecord that never sees this repair on its own — must feed a non-None result into CompositionRecord.seed_pages_from_array itself, or it re-derives entries from the still-corrupted on-disk bytes.

Return type:

tuple[list[Finding], bytes | None]

async synology_apm_repo.sdk.dedup.verify_checks.check_bucket_structure(store, reader)

Everything about this bucket that doesn’t depend on which chunk(s) a caller actually cares about: expected_bucket_size against the real on-disk size, and the ChunkCrcStore trailer’s own crcOfChunkCrc self-consistency (via BucketReader.ensure_chunk_crc_store, which also warms the cache check_chunk_ciphertext_crc below reuses — one trailer read shared by both, not two). Header/SizeStore CRC are already unconditionally checked by BucketReader.open itself; the one further thing owned here is turning a successful SizeStore self-repair (reader.sizestore_repaired) into its own Symptom.REPAIRED_VIA_PARITY finding, since open() itself has no Finding-returning contract — the same visibility check_map_and_attr_crc already gives the composition-layer map-CRC repair.

A legacy uncompressed-layout bucket has neither an expected_bucket_size formula nor a ChunkCrcStore trailer that applies to it — both are skipped, not treated as findings.

Takes a bare store, not a full DedupRepo — the only thing this function ever needed from one — so a caller with just an ObjectStore in hand (a multiprocess worker rebuilding its own repository state via dedup.pool_descriptor, which has no DedupRepo at all) can call this directly.

async synology_apm_repo.sdk.dedup.verify_checks.check_chunk_ciphertext_crc(reader, chunk_idx)

One chunk’s stored bytes against its own ChunkCrcStore entry (FORMAT-SPEC.md: ChunkCrcStore) — no decrypt/decompress attempt, so this needs no vault key and runs the same whether or not the bucket turns out to be encrypted.

async synology_apm_repo.sdk.dedup.verify_checks.check_raw_chunk_ciphertext_crc(reader, chunk_idx, raw)

check_chunk_ciphertext_crc’s counterpart for a caller that already has chunk_idx’s stored bytes in hand — e.g. from a batched read_raw_chunks call — instead of fetching them itself, the same split verify_raw_chunk_ciphertext_crc already has from verify_chunk_ciphertext_crc. Same three-way exception mapping as the singular form above; kept as its own function rather than folded into check_chunk_ciphertext_crcs below (its only caller) so that caller’s own batched-fetch-then-check-each shape stays readable.

async synology_apm_repo.sdk.dedup.verify_checks.check_chunk_ciphertext_crcs(reader, chunk_indices)

Batched check_chunk_ciphertext_crc: every one of chunk_indices’s stored bytes is fetched via reader.read_raw_chunks — one merged store.read() per contiguous run instead of one per chunk — before checking each against its own ChunkCrcStore entry via check_raw_chunk_ciphertext_crc. Reports each chunk that fails on its own, the same granularity looping check_chunk_ciphertext_crc would.

read_raw_chunks reads its own runs one after another with no per-run isolation: a later run’s failure raises past every earlier run’s already-fetched bytes, discarding them along with it. Since those bytes were never actually checked, silently swallowing that failure here would mean a real ciphertext CRC mismatch sitting in an earlier, otherwise-successful run could go unreported — so any failure at all from the batched read falls back to one check_chunk_ciphertext_crc call per chunk instead, which has no such cross-run coupling. Reserved for the failure path only; the common case (a single run, or every run succeeding) never pays for it.