Software design

Software design#

General#

The design realizes the reverse chain: it inverts the operational L0→L1B radiometric chain from Sentinel-2B L1B DN (offset → relative-response/PRNU → dark → un-bin → SWIR re-stage → defective → crosstalk → on-board-eq) via forward_radiometric_atbd.reverse_l1b_to_l0, reconstructing L1A → L0plus (CCSDS-122 ISP) → Synthetic L0. The inversion is orchestrated by the run_pipeline phases reverse-l1b → package-l0 → validate-reverse, with MTF-deconvolution OFF so PSF and noise are not re-applied, and validated against the reference ESA L0 img (10/20 m bands ≤~4 DN). The package is layered: a constants/sensor leaf, an ADF/GIPP layer, the chain, and a product-assembly integrator. Every algorithm is implemented from the public L1 ATBD and the GIPP data format.

Overall architecture#

Key design decisions#

  • Official ATBD radiometric model \(X = A \cdot G \cdot L + D\). The reverse chain takes the operational relation — relative response G, dark D, on-board equalization and offset — only as the model it inverts in the downlink DN domain from S2 L1B. It is the exact conjugate inversion of that chain, not a forward generator, and it does not apply the forward radiance→DN absolute multiply.

  • Real per-pixel dark + relative response from the operational GIPP (R2EQOG COEFF_D / cubic (A,B,C) / bilinear (A1,A2,Zs)); no fitted or seeded values in the referenceized path — these are the coefficients the reverse chain inverts.

  • Calibration sub-set (inverse-crime cure). The downstream calibration coefficients are derived independently from synthetic CSM sun-diffuser + dark acquisitions, rather than reusing the exact ADF values the reverse chain consumed for its inversion.

  • Original, self-contained. No external processor is installed, imported, or named; the forward model comes from the public L1 ATBD and the calibration coefficients from the GIPP data files only.

Software components design — General#

sensor.py — sensor model (REQ-FUNC-003)#

Real harvested S2 constants and the Band dataclass. Key API: BANDS, UNITS, band(), all_bands(), band_number(), zarr_band_key(), spectral_band_info(), unit_from_platform(); Band properties .dn_ref, .cal_gain, .dark_dsnu. Carries PHYSICAL_GAIN, LREF, SNR_AT_LREF, NOISE_ALPHA/BETA, TDI_BANDS, SWIR_BANDS, DN_MAX, LINE_PERIOD_MS, NUC_TABLE_ID, RADIO_ADD_OFFSET_L1B, DARK_PEDESTAL_LSB, EQ_GAIN_STD.

gipp.py — operational GIPP parser (REQ-FUNC-046, -015, -019, -012, -017, -018)#

Original xml.etree parser → per-pixel arrays. API: DetectorEq (.rel_gain), BandEq (.npix), RadioParams, GippSet; read_r2eqog_band(), read_r2depi(), read_blindp(), read_r2para(), read_r2crco(), load_gipp_set(). EOPF ADF parsers for the full reverse chain: read_r2eqog_eopf() (dark+PRNU from ADF_REQOG), read_rswir_eopf() (SWIR staggered-readout shift map

  • B10 kernel, ADF_RSWIR), read_reob2_eopf() (on-board-eq a1/a2/zs/d, ADF_REOB2), read_rcrco_eopf() (13×13 crosstalk, ADF_RCRCO); GippSet carries the ADF paths and reads swir_shift() / onboard_eq() lazily.

adf.py — ADF assembly (REQ-FUNC-044, -046; REQ-IF-003)#

BandADF frozen dataclass with from_gipp() (per-pixel dark + PRNU, blind-column width alignment), from_product() (L1B-derived), synthesize() (fallback) — the reverse chain’s ADF inputs.

forward_radiometric_atbd.py — reverse chain + operational-eq inversion (REQ-FUNC-010, -011, -013, -015, -016, -020, -022)#

Full S2 L1B→Synthetic L0 reverse (the reverse chain): reverse_l1b_to_l0() inverts the full radiometric chain in the downlink DN domain (offset → G⁻¹ (relative-response/PRNU) → on-board eq → dark → un-bin → SWIR re-stage → defective), with inverse_equalize() / forward_equalize() (on-board-eq inversion — cubic via Newton, bilinear closed-form) and helpers reapply_onboard_eq() (REOB2 bilinear non-linearity) and restage_swir_lines() (S8 whole-line roll / B10 sub-pixel kernel). The reconstructed DN is clipped and rounded to the 12-bit uint16 product scale inline (np.clip/np.rint, REQ-FUNC-022). Restoration/deconvolution (fwd step 8) is off, so PSF re-blur / noise are not re-applied.

reverse.py — exact-inverse bridge (retained module)#

Thin module retained and imported at package load. It exposes the exact-inverse bridge reverse_radiometric() / forward_radiometric() between the DN and L1B domains. The operational radiometric inversion of S2 L1B is performed by forward_radiometric_atbd.reverse_l1b_to_l0 and gipp (see above), not by a per-step forward-impress chain.

calibration.py — calibration sub-set (REQ-FUNC-047; REQ-PERF-004)#

DerivedCalibration; synth_dark_acquisition(), synth_diffuser_acquisition(), derive_dark(), derive_relative_response(), calibrate(), estimated_adf().

isp.py — CCSDS ISP / telemetry (REQ-FUNC-030, -092)#

build_primary_header(), parse_primary_header(), cuc_time(), apid_for(), packetize_stream() / iter_packets() / reassemble_segments() (compressed payload transport, SEQ_FIRST/CONT/LAST grammar), build_sad_packets(); frame_isp_headers() is legacy.

ccsds122.py — CCSDS 122.0-B lossless image compression (REQ-FUNC-092; ICD-IF-C122)#

compress_frame() / decompress_frame() (bit-exact), dwt97m_forward() / dwt97m_inverse(), parse_segment_headers(), segment_byte_bounds(), CompressionStats.

naming.py — EOPF PSFD §3 product naming (REQ-FUNC-091; ICD-IF-NAME)#

psfd_name(), parse_psfd_name() (exact inverse), from_l1a_context() (flagged fallbacks).

s3fetch.py — anonymous S3 fetch (REQ-FUNC-093 input path)#

list_prefix() (paginated), fetch_prefix() (verified parallel GET, resume, traversal guard), parse_list_xml(), save_manifest().

io.py — product reader (REQ-IF-001, REQ-FUNC-001)#

read_l1b_band(), read_l1a_raw().

l0product.py — Synthetic L0 RAW assembly (REQ-FUNC-030..-034, -045; REQ-IF-002)#

reverse_to_l0_frames(), build_root_metadata(), write_l0_product() (L0plus — CCSDS ISP + ancillary, eopf_type selectable), write_l0_decoded_product() (decompressed-img L0 mirroring the reference S2B S02MSIL0__ layout — measurements/d{DD}/b{BB}/img + decode-quality attrs, for a direct ESA-L0 compare), read_l0_isp_dn().

import_l0.py — L1A materialisation + public-L0 import (REQ-FUNC-001, -045)#

read_public_l0_identity(), convert() (public Synthetic L0 → PDI-style L1A), write_l1a_product() — the reverse chain’s materialised L1A writer: multi-detector raw counts → measurements/DD{dd}/{BAND}/l1a_raw_image

  • L1A STAC, serial (NFS-safe) with a bit-identical re-read assert. Consumed back by io.read_l1a_raw().

The reverse chain (run_pipeline phases) is reverse-l1b → package-l0 → validate-reverse: reconstruct S2 L1B → materialise L1A, then L1A → L0plus (ISP) → Synthetic L0 (decoded img), then compare the Synthetic L1A against the reference S2B reference ESA L0 img (framing-aligned; no decoding — the archived EOPF L0 already stores decompressed img, verified on the 2024-04-08 TC7D granule).

Software components design — Aspects of each component#

All components are stateless functions or frozen dataclasses operating on NumPy arrays; the only I/O is in io.py (read) and l0product.py (write). Coefficient provenance is carried on BandADF.source and in the Synthetic L0 adf_provenance metadata (REQ-FUNC-045).

Internal interface design#

Arrays are 2-D (lines, detector_columns); per-pixel coefficients are (act,) broadcast over lines. BandADF is the contract object between adf/gipp and the reverse chain (forward_radiometric_atbd); DetectorEq between gipp and forward_radiometric_atbd; FrameKey = tuple[int, str] (detector, band) keys the L0 frame dict in l0product. See the ICD (docs/icd.md) for the external product interfaces.