synology_apm_repo.sdk.format.redundancy module

Redundancy blob sizing and self-repair (FORMAT-SPEC.md: ChunkCrcStore).

Every Composition record and every .buk bucket carries a trailing Redundancy blob, written unconditionally on every flush: a 2*coverage -byte ring-buffer XOR parity (windows ping-ponged across two halves — every even-indexed coverage-byte window XORs into parity[0:coverage], every odd-indexed window into parity[coverage:2*coverage]) plus a rolling (cumulative, not independent-per-window) CRC32 checkpoint after each window (StepCrc).

This module owns both halves of that blob’s story:

• redundancy_size — the size formula, load-bearing for every reader (needed to compute a bucket’s/composition record’s total on-disk size independently of trusting the file’s own declared length — see expected_bucket_size and composition.py’s record_total_length).

• parse_redundancy_blob/attempt_repair — the actual self-repair algorithm. Production servers write this blob but never read it back (ordinary reads only ever validate the plain CRC the blob’s own data argument below is checked against — see this SDK’s own verify_chunk_map_crc/parse_size_store callers); this SDK is the first real consumer of it, since it can do so entirely in-memory and never needs to write a repaired copy back to the source repository. repair_via_trailer wraps it with the “fetch the trailer, then attempt_repair” shape every call site otherwise duplicated.

Because StepCrc is rolling, not independent per window, this mechanism can locate and repair at most one corrupted region, bounded to two consecutive coverage-windows — a “single-disk-failure RAID” limit, not N-way independent parity (FORMAT-SPEC.md: ChunkCrcStore). See attempt_repair for how it confirms a repair before returning it.

synology_apm_repo.sdk.format.redundancy.redundancy_size(data_size, coverage)

Total byte length of a Redundancy blob covering data_size bytes of underlying data, checkpointed every coverage bytes:

16 (header) + 4 * ceil(data_size / coverage) (StepCrc array) + min(data_size, 2 * coverage) (parity) (FORMAT-SPEC.md: ChunkCrcStore).

coverage is 256 for bucket trailers, 8192 for composition record trailers (FORMAT-SPEC.md: ChunkCrcStore).

class synology_apm_repo.sdk.format.redundancy.RedundancyBlob(coverage, data_size, step_crc, parity)

Bases: object

One parsed Redundancy trailer: the rolling per-window CRC32 checkpoints (step_crc, one per coverage-byte window of the protected data, the last window truncated if data_size isn’t an exact multiple of coverage) plus the 2*coverage-byte XOR parity ring buffer (parity) — the even/odd-window-into-each-half layout described above, that attempt_repair reconstructs from.

coverage: int

data_size: int

step_crc: tuple[int, ...]

parity: bytes

synology_apm_repo.sdk.format.redundancy.parse_redundancy_blob(data, *, data_size, coverage)

Parse a Redundancy trailer of exactly redundancy_size(data_size, coverage) bytes (FORMAT-SPEC.md: ChunkCrcStore) — the raw magic/ version/coverage/data-size header fields are cross-checked against the caller-supplied data_size/coverage (the values already known independently, from the record’s own RecordHead/bucket header), not trusted blindly from the blob itself.

Raises:

FormatError – data is shorter than expected, or its magic/ version/coverage/data_size header fields don’t match what was expected — either way, this Redundancy blob cannot be used for repair (attempt_repair treats this as “no repair possible”, not a hard failure).

synology_apm_repo.sdk.format.redundancy.attempt_repair(data, redundancy_raw, *, coverage, expected_crc)

Best-effort, in-memory self-repair of data (whose CRC32 is already known not to match expected_crc) using its trailing Redundancy blob (FORMAT-SPEC.md: ChunkCrcStore).

Reconstructs the pair of consecutive windows [bad_idx, bad_idx+1] around the first divergent rolling checkpoint (one window from each parity half, since the rolling checkpoint alone can’t distinguish “only bad_idx is corrupted” from “bad_idx+1 might be too”) and re-validates the whole candidate reconstruction’s own CRC32 against expected_crc before ever returning it — so a non-None result is byte-for-byte confirmed correct, never a guess.

Returns:

The fully repaired data (same length) on a confirmed-correct reconstruction; None if the blob itself doesn’t parse, the rolling scan can’t localize a starting point, or the candidate reconstruction still doesn’t validate (more than one region corrupted, or the Redundancy blob itself is also damaged). Never raises — every failure mode collapses to None, since a caller’s own fallback (propagate the original DataCorruptError unchanged) is identical either way.

Return type:

bytes | None

async synology_apm_repo.sdk.format.redundancy.repair_via_trailer(data, *, coverage, expected_crc, fetch_trailer)

Shared orchestration behind every Redundancy-blob self-repair call site (dedup/pool/_bucket_reader.py’s SizeStore repair, dedup/verify_checks.py’s map-CRC repair): fetch the trailer via fetch_trailer — already resolved to that record’s/bucket’s own trailer offset, however that offset needs computing at each call site — and hand it to attempt_repair.

attempt_repair runs on a real OS thread (asyncio.to_thread), the same responsiveness rationale as dedup/pool/_bucket_reader.py’s _read_run: the map-CRC repair path can see an array up to several MB, and its rolling CRC32 scan plus reconstruction must not stall every other concurrent Task for that duration.

Returns:

attempt_repair’s own result, or None if fetch_trailer itself raises NotFoundError/FormatError — the “trailer can’t be fetched” case every call site otherwise handled with its own identical try/except.

Return type:

bytes | None