msi_processor.computing.l0_decode package

Level-0 decode stage (C-PU-L0; DPM-M-L0; ALG-L0-DEC, ALG-L0-LOSS).

Turns the downlinked Level-0 (L0c) product into L1A: detector DN in focal-plane geometry, passed-through acquisition telemetry (timing, orbit, attitude), and seeded QA, with deterministic trailing line-loss truncation. The stage is a pure core plus a thin L0DecodeUnit wrapper. The sensor-private source-packet decode body is held outside this public distribution; the public path operates on the documented open-container sample layout.

Submodules

msi_processor.computing.l0_decode.ccsds122 module

CCSDS 122.0-B image-data compression — lossless subset (integer DWT + bit-plane coder).

Note

Consumer-side copy. This module is the same public CCSDS 122.0-B codec the producer (s2-msi-raw-generator) implements; the consumer carries its own copy so the ground decode (the operational L1A-side decompression) lives in the processing facility. Both copies are kept bit-identical (Apache-2.0, same project family).

Sentinel-2 compresses MSI video data onboard with the proprietary MRCPB wavelet scheme (bit-plane coding, “similar to JPEG 2000”); a CCSDS-compression ASIC is the documented alternative option (eoPortal, S2 CoReCi). This module implements that alternative — CCSDS 122.0-B — in its lossless profile, pure numpy, so the generator’s L0 ISP payloads carry genuinely wavelet-compressed image data and the ground segment (L1A side) can restore the exact DN.

Implemented per the Blue Book structure

• Integer DWT 9/7-M (CCSDS 122.0-B §3.3): three 2-D decomposition levels, lifting form with the specified floor(·+1/2) rounding and whole-sample symmetric extension; exactly invertible on integers.

• Block/family structure (§4.1): one 8×8 pixel area → 1 DC + 63 AC coefficients in three families (parent, 2×2 children, 4×4 grandchildren per family), raster scan; gaggles of 16 blocks; segments of segment_blocks blocks.

• Segment headers (§4.2): the Part-1A/3/4 semantic content (start/end flags, segment count, BitDepthDC/BitDepthAC, segment size, DWT type, pixel bit depth, image dimensions) is carried in an explicit little-endian layout (see parse_segment_headers); the stream is fully self-describing.

• DC coefficient coding (§4.3): per-segment DPCM against a raw reference sample, zigzag signed→nonnegative mapping, per-gaggle Rice coding with brute-force parameter selection and an uncoded escape option.

• Per-block BitDepthAC (§4.4): same DPCM + per-gaggle Rice machinery.

Documented divergences (kept deliberately small, listed in the ICD/ATBD)

1. §4.5.3 word mapping / variable-length codes are not implemented. AC bit-plane coding keeps the stage semantics and scan order (significance → sign → refinement, plane-sequential from BitDepthAC−1 down to 0, block-major within a segment) but the per-stage bits are packed raw (numpy.packbits) instead of entropy-coded words. Consequence: bit-exact lossless with the full segment/gaggle/block/plane structure, but the stream is not interoperable with reference CCSDS-122 decoders (the matching decoder lives in this module) and ratios run below a full BPE.

2. Header field packing uses explicit byte-aligned little-endian fields (with section byte-lengths) rather than the Blue Book bit layout; the content mirrors Parts 1A/3/4.

3. Sections are byte-aligned; the Blue Book packs them contiguously.

The API is frame-oriented: compress_frame() → (payload bytes, CompressionStats) and decompress_frame() → the exact uint16 DN frame. Segments default to one block row (8 image lines) so packetization downstream maps segments to CCSDS space-packet groups with line-accurate datation.

class msi_processor.computing.l0_decode.ccsds122.CompressionStats(raw_bytes, compressed_bytes, n_segments, n_blocks, pixel_bit_depth, dc_bytes, bitdepth_ac_bytes, ac_bytes, header_bytes)

Bases: object

Byte accounting of one compressed frame.

Attributes:

bits_per_pixel

Compressed bits per pixel.

ratio

Raw ÷ compressed size (packed-sample raw, pixel_bit_depth bits/px).

ac_bytes: int

bitdepth_ac_bytes: int

property bits_per_pixel: float

Compressed bits per pixel.

compressed_bytes: int

dc_bytes: int

header_bytes: int

n_blocks: int

n_segments: int

pixel_bit_depth: int

property ratio: float

Raw ÷ compressed size (packed-sample raw, pixel_bit_depth bits/px).

raw_bytes: int

msi_processor.computing.l0_decode.ccsds122.GAGGLE_BLOCKS = 16

Blocks per gaggle (CCSDS 122.0-B §4.1).

msi_processor.computing.l0_decode.ccsds122.MAGIC = b'C122LSv1'

Frame-stream magic (8 bytes) + container version.

msi_processor.computing.l0_decode.ccsds122.compress_frame(dn, *, pixel_bit_depth=12, segment_blocks=None)

Losslessly compress a 2-D DN frame → (payload, stats).

segment_blocks defaults to one block row (width // 8), aligning each segment with 8 image lines. The payload is fully self-describing (see module docstring).

Return type:

tuple[bytes, CompressionStats]

msi_processor.computing.l0_decode.ccsds122.decompress_frame(payload)

Exact inverse of compress_frame() → uint16 frame.

Return type:

ndarray

msi_processor.computing.l0_decode.ccsds122.dwt97m_forward(frame, levels=3)

3-level (default) 2-D integer 9/7-M DWT → subband dict LL<n>, HL<l>, LH<l>, HH<l>.

Rows are transformed first, then columns; the recursion descends into LL. Input dimensions must be divisible by 2**levels.

Return type:

dict[str, ndarray]

msi_processor.computing.l0_decode.ccsds122.dwt97m_inverse(bands, levels=3)

Exact inverse of dwt97m_forward().

Return type:

ndarray

msi_processor.computing.l0_decode.ccsds122.parse_segment_headers(payload)

Header inventory of a compressed stream (frame fields + per-segment Part-1A content).

Return type:

dict[str, Any]

msi_processor.computing.l0_decode.ccsds122.segment_byte_bounds(payload)

Byte offsets of packetization groups: [0, seg1_off, …].

The frame header is folded into the first group, so joining the groups reproduces the exact stream — the property decompress_frame() needs after packet reassembly.

Return type:

list[int]

msi_processor.computing.l0_decode.core module

Pure Level-0 decode core (C-PU-L0; ALG-L0-DEC, ALG-L0-LOSS).

CPM-free, I/O-free numpy functions implementing the public Level-0 → L1A scaffolding of ATBD <5.1>: deterministic line-loss detection and truncation (ALG-L0-LOSS), structural legality checking (REQ-F-L0-03) and initial QA flagging. Arrays are 2-D (line, detector) in focal-plane geometry.

The sensor-private source-packet reassembly / decompression (ALG-L0-DEC) is a profile-bound [impl] backend held outside this public distribution; the public path here operates on already-decoded open-container detector frames (the documented sample layout the simulator and the operational decoder both produce). decode_source_packets() is the explicit fail-stop seam.

Trace: REQ-F-L0-01..05; DPM-M-L0; ALG-L0-DEC, ALG-L0-LOSS.

class msi_processor.computing.l0_decode.core.L0DecodeParams(bit_depth=12, line_factor=<factory>, fill_value=None, max_lost_fraction=None)

Bases: object

Tunable Level-0 decode parameters (SDD <5.4.2>; DPM-PRM-L0-01).

Parameters:

bit_depth:

Sensor radiometric depth; legal DN range [0, 2**bit_depth - 1] (DPM-PRM-GEN-01, default 12).

line_factor:

Per-band line decimation factor echoed for provenance (e.g. a PAN band acquired at twice the line rate); does not transform the samples here.

fill_value:

No-data sentinel DN; flagged NO_DATA when present.

max_lost_fraction:

Legality bound (REQ-F-L0-03): reject a band whose truncated line loss exceeds this fraction. None disables the bound.

Attributes:

fill_value

max_dn

Upper bound of the valid DN range, 2**bit_depth - 1.

max_lost_fraction

bit_depth: int = 12

fill_value: Optional[int] = None

line_factor: Mapping[str, int]

property max_dn: int

Upper bound of the valid DN range, 2**bit_depth - 1.

max_lost_fraction: Optional[float] = None

msi_processor.computing.l0_decode.core.check_legality(frames, params)

REQ-F-L0-03 — validate decoded frame structure / DN range; raise on fault.

Checks that the frame set is non-empty, every band is a 2-D integer-valued frame within [0, max_dn], and (if max_lost_fraction is set) that the trailing line loss is within budget. Raises InputValidationError.

Return type:

None

msi_processor.computing.l0_decode.core.decode_source_packets(raw, codec_spec)

ALG-L0-DEC — reassemble/decompress source packets to detector frames.

The documented open downlink form — the producer’s canonical L0 (measurements/d{DD}/b{bb}/isp: CCSDS space packets carrying CCSDS-122 lossless payloads) — is ground-decoded here bit-exactly (ground_decode, REQ-F-L0D-06): the real-chain L1A-side decompression now lives in the consumer.

Any other on-wire packetisation remains sensor/NDA-specific and profile-bound: that body is a private [impl] backend not part of this public distribution, and fail-stops below.

Return type:

dict[str, ndarray[tuple[Any, ...], dtype[uint16]]]

msi_processor.computing.l0_decode.core.detect_line_loss(band)

ALG-L0-LOSS — boolean mask of trailing lost lines in a band.

Deterministic rule (ATBD <5.1>): downlink line loss zero-fills the tail of the frame, so the loss boundary is the first all-zero line that follows a non-zero line; every line from there to the end is marked lost. A frame with no non-zero→zero transition has no loss. Interior partially-corrupted (non-zero) lines are not caught here (ATBD <5.1> open point).

Return type:

ndarray[tuple[Any, ...], dtype[bool]]

Parameters:

band:

2-D (line, detector) DN frame.

Returns:

BoolArray

Per-line mask, True on lost lines.

msi_processor.computing.l0_decode.core.initial_qa(band, params)

Build the initial L1A QA mask for a (truncated) band (REQ-F-QA-02 seed).

Sets NO_DATA on fill-value pixels (if a sentinel is configured) and on any wholly-zero interior line. Bits accumulate monotonically downstream (OR-only); the initial mask is otherwise 0.

Return type:

ndarray[tuple[Any, ...], dtype[uint16]]

msi_processor.computing.l0_decode.core.truncate_loss(band, loss_mask)

Drop the trailing lost lines (REQ-F-L0-02); return (kept, n_lost).

Return type:

tuple[ndarray[tuple[Any, ...], dtype[Any]], int]

msi_processor.computing.l0_decode.ground_decode module

ALG-L0-DEC — the consumer-side ground decode of the canonical (downlink) Level-0.

The real-chain L1A-side operation (the Sentinel-2 L0→L1A relation is decode/packaging: L0 stores compressed instrument source packets, L1A decompresses): walk the CCSDS space packets of a band’s isp stream, reassemble the segment groups (seq-flags grammar + 14-bit counter continuity enforced), join them back into the CCSDS-122 stream and decode it bit-exactly to the detector DN frame.

This implements the documented open interface of the producer’s canonical L0 (measurements/d{DD}/b{bb}/isp — ICD-IF-ISP / ICD-IF-C122 in the s2-msi-raw-generator ICD). Truly sensor-private on-wire formats still fail-stop in decode_source_packets().

numpy-only (plus zarr for the path-based helper); no eopf dependency.

msi_processor.computing.l0_decode.ground_decode.decode_canonical_frames(band_groups)

Decode a mapping of canonical band groups → {BAND: DN}.

band_groups maps d{DD}/b{bb} paths to objects exposing an isp member (zarr groups or nested mappings of arrays). Band keys are upper-cased to the processor convention (b8a → B8A).

Return type:

dict[str, ndarray]

msi_processor.computing.l0_decode.ground_decode.decode_canonical_l0(path, detector, band)

Path-based helper: canonical L0 zarr → one band’s bit-exact DN frame.

Return type:

ndarray

msi_processor.computing.l0_decode.ground_decode.decode_stream(stream)

One band’s canonical isp stream → the exact uint16 DN frame (bit-exact).

Return type:

ndarray

msi_processor.computing.l0_decode.ground_decode.iter_packets(buf)

Iterate CCSDS packets in a concatenated stream → (header, cuc_seconds, body).

The primary header’s Packet Data Length field walks the stream; a stream is well-formed iff packets tile it exactly. body excludes the CUC secondary header.

Return type:

Iterator[tuple[dict[str, int], Optional[float], bytes]]

msi_processor.computing.l0_decode.ground_decode.parse_primary_header(b)

CCSDS Space Packet primary header → field dict (decode direction).

Return type:

dict[str, int]

msi_processor.computing.l0_decode.ground_decode.reassemble_segments(buf)

Packets → per-segment byte streams (strict grammar + continuity).

Enforces the seq-flags grammar (FIRST → CONT* → LAST, or STANDALONE) and the 14-bit sequence-counter continuity; raises ValueError on gaps or malformed sequences.

Return type:

list[bytes]

msi_processor.computing.l0_decode.unit module

Thin EOProcessingUnit wrapper for Level-0 decode (C-PU-L0).

Adapts the pure core to the EOPF CPM runtime (SDD <5.4.2>): read parameters, obtain decoded detector frames, detect and truncate line loss, seed QA, pass acquisition telemetry through, and build the L1A EOProduct. No algorithm lives here.

Decode seam (SDD <5.4.2>). The on-wire source-packet reassembly / decompression (ALG-L0-DEC) is sensor-private and held outside this public distribution. The unit therefore takes one of two L0c forms:

• an open container that already exposes decoded samples under measurements/detector/<band> (the documented sample layout the simulator and the operational decoder both emit) — the public path exercised here;

• an undecoded on-wire form — which routes to decode_source_packets(), the private [impl] backend, and fail-stops with L0DecodeError when that backend is absent.

Trace: REQ-F-L0-01..05; DPM-M-L0; ALG-L0-DEC, ALG-L0-LOSS; ICD <5.3.1>A.

class msi_processor.computing.l0_decode.unit.L0DecodeUnit(identifier='')

Bases: EOProcessingUnit

Level-0 decode processing unit (C-PU-L0; SDD <5.4.2>).

Attributes:

identifier

Identifier of the processing step

Methods

run:	Decode (open-container) L0c into an L1A product: detector DN in focal-plane geometry, passed-through acquisition telemetry, and seeded QA, with trailing line loss truncated (ALG-L0-LOSS).

PROCESSOR_LEVEL = 'L1A'

PROCESSOR_MODEL = True

PROCESSOR_NAME = 'msi_l0_decode'

PROCESSOR_VERSION = '1.1.0'

run(inputs, adfs=None, mode=None, **kwargs)

Decode an L0c product into L1A.

Return type:

Mapping[str, Union[EOProduct, EOContainer, DataTree, Iterable[Union[EOProduct, EOContainer, DataTree]]]]

Parameters:

inputs:

{"l0c": EOProduct} — the Level-0 product (open-container form carries measurements/detector/<band> + conditions/*).

adfs:

None — no auxiliary data is consumed at Level-0 (REQ-F-L0-01).

mode:

"nominal".

**kwargs:

L0DecodeParams fields plus optional name for the output.

Returns:

Mapping[str, DataType]

{"l1a": EOProduct} — detector DN, conditions, and l0_flags.