s2_msi_raw_generator package

s2_msi_raw_generator package#

Sentinel-2 MSI reverse E2ES — degrade L1A/L1B radiance back to synthetic Synthetic L0 RAW.

The reverse / forward-instrument conjugate of msi-processor. The chain is radiometric-only (14-step chain, L1A/L1B in per-detector geometry → Synthetic L0 RAW DN); an L1C-entry + geometry-reverse module was cancelled — L1A/L1B is already in detector geometry, so there is nothing to de-orthorectify (Issue #17). See docs/atbd/atbd.md.

Submodules#

s2_msi_raw_generator.adf module#

Per-band ADFs for the reverse chain — published data where it exists.

Provenance of each component:

All values are used verbatim fromS2 sources — nothing is fitted. The raw model is the official L1 ATBD equation X = A·G·L + D (S2-PDGS-MPC-ATBD-L1 §4.1.1). Provenance per component:

gain (A) — Band.cal_gain, the absolute calibration A in S1 (DN=A·L), derived from the reference
noise α,β + SNR@Lref so the chain reproduces SNR@Lref. The product’s physical_gain is incoherent with α,β on this synthetic dataset (it mis-scales low- radiance bands by up to ~10×), so it is kept only for L0 metadata / the round-trip bridge.
PSF — REAL official ESA per-band, per-unit PSF matrices (SentiWiki S2{A,B,C}_PSF.zip,
data/psf/); 33×33 oversampling-5 matrices integrated to the detector grid. B10 has no published PSF (water-vapour band) → identity kernel. See real_psf_kernel.
spectral — REAL per-unit centre/bandwidth/equivalent wavelength (SRF doc, in sensor.py).
noise α,β — REAL noise model σ=√(α²+β·DN) (S2-RUT) with α, β straight from the L1A product
metadata (sensor.NOISE_ALPHA/NOISE_BETA), used verbatim. With the cal_gain DN scale the chain reproduces the SNR@Lref.
dark (D), PRNU (G) — REAL per-pixel values from the operational S2A GIPP R2EQOG (dark COEFF_D
≈440–522 LSB; relative-response gains cubic A/B/C / bilinear A1/A2/Zs), parsed by s2_msi_raw_generator.gipp and built via BandADF.from_gipp. Fallbacks: the Feb-2023 DQR dark (sensor.DARK_PEDESTAL_LSB + Band.dark_dsnu) and BandADF.from_product (L1B-derived PRNU); synthesize seeds representative values when no GIPP is supplied. Onboard-eq gain uses the measured stability (0.05 % 1σ, no offset; sensor.EQ_GAIN_STD).

class s2_msi_raw_generator.adf.BandADF(band, noise_a, noise_b, psf, prnu_gain, dark_dn, eq_gain, eq_offset, prnu_is_real=False, source='synthetic')#

Bases: object

Per-band ADF set for the reverse chain (per detector of width n_det).

psf and the band’s spectral/gain values are (published). The per-detector prnu_gain/dark_dn/eq_* arrays are either product-derived (from_product) or seeded representative values (synthesize) when the operational GIPP is not supplied.

band: Band#

dark_dn: ndarray#

eq_gain: ndarray#

eq_offset: ndarray#

classmethod from_gipp(b, detector, gippset, *, active_width=None)#

Build a BandADF for one detector from the REAL operational GIPP (R2EQOG): per-pixel dark signal D and relative-response (PRNU) gain (C cubic / A1 bilinear).

gippset is a s2_msi_raw_generator.gipp.GippSet. When active_width is given and differs from the GIPP across-track size, the blind columns (from BLINDP) are stripped so the arrays align to the active product width. PSF and noise stay (SentiWiki PSF, product noise model).

Return type:: BandADF

classmethod from_product(b, *, prnu_gain, dark_dn, eq_gain=None, eq_offset=None)#

Build a BandADF from REAL per-detector PRNU/dark arrays (e.g. derived from the matched L0↔L1A products by scripts/derive_prnu_dark.py). PSF and noise stay ESA-sourced.

Return type:: BandADF

noise_a: float#

noise_b: float#

prnu_gain: ndarray#

prnu_is_real: bool = False#

psf: ndarray#

source: str = 'synthetic'#

s2_msi_raw_generator.adf.fit_noise_coeffs(b, read_fraction=0.1)#

Fallback fit of (a, b) of σ² = a + b·DN to SNR@Lref — used only if the product noise model is unavailable. Prefer noise_coeffs() (the α, β).

Splits the total variance at Lref (σ_ref² = (DN_ref/SNR)²) into a read/dark floor a = read_fraction·σ_ref² and a shot term b = (1-read_fraction)·σ_ref²/DN_ref, so that σ(DN_ref) = σ_ref exactly (reproduces SNR@Lref) with a, b > 0 for every band.

Return type:: tuple[float, float]

s2_msi_raw_generator.adf.gaussian_psf(mtf_nyquist=0.25, radius=4)#

Normalized 2-D separable Gaussian PSF whose discrete MTF at Nyquist == mtf_nyquist.

The continuous formula MTF(f)=exp(-2π²σ²f²) overestimates attenuation for a sub-pixel, truncated, re-normalized kernel, so σ is calibrated numerically (bisection) to hit the discrete target. Kernel sums to 1 (DC gain 1, radiometry-preserving).

Return type:: ndarray

s2_msi_raw_generator.adf.load_oversampled_psf(band, unit='S2A')#

Load the 33×33 oversampled PSF matrix for band/unit; None if none published.

B10 (water-vapour band) has no published PSF — returns None. Result is cached and read-only.

Return type:: ndarray | None

s2_msi_raw_generator.adf.noise_coeffs(b)#

REAL noise model coefficients (a, b) for σ = √(a + b·DN) from the L1A product.

The official S2-RUT model (Gorroño & Gascon) is Noise(DN) = √(α² + β·DN), with α, β straight from the product’s quality_indicators_info/…/noise_model (sensor.NOISE_ALPHA/NOISE_BETA). Returns a = α² and b = β so the chain’s σ=√(a+b·DN) is exactly the RUT model; it reproduces the spec SNR@Lref. No fitting.

Return type:: tuple[float, float]

s2_msi_raw_generator.adf.real_psf_kernel(band, unit='S2A', oversampling=5)#

Real detector-grid PSF kernel for band/unit (Σ=1, DC gain 1).

The published PSF is oversampled ×``oversampling``; integrate each oversampling``×``oversampling block (centred on the matrix centre) to the detector-pixel grid. B10 → identity (no re-blur).

Return type:: ndarray

s2_msi_raw_generator.adf.synthesize(b, n_det, *, seed=0, prnu_std=0.005)#

Build a BandADF for band b over n_det detector columns.

PSF (ESA, per-unit), spectral, gain and the noise model (α, β from the product) are all ESA-sourced. The per-detector PRNU/dark/equalization arrays are seeded representative values — use BandADF.from_gipp() with the (publicly fetchable) operational GIPP for the reference per-pixel coefficients, or BandADF.from_product() for product-derived arrays; this synthetic fallback exists for GIPP-less environments only. (The early “credentialed GIPP” blocker #36 is resolved — see ATBD §8.)

Return type:: BandADF

s2_msi_raw_generator.adf_writer module#

Serialize derived calibration coefficients to EOPF-CPM Auxiliary Data Files (the “cal-DB”).

Option Y of the generator ⇄ msi-processor coupling: the Synthetic Raw Data Generator derives realistic radiometric calibration coefficients and writes them as versioned EOPF zarr ADFs that the downstream processor (the L1PP blocks inside msi-processor) consumes directly as its nuc / dark / radiometric Auxiliary Data Files. Nothing here re-implements the processor; it only produces the ADF set the processor already expects (single shared sensor-model ADF — the e2es-coupling-decision ADR).

ADF schema — matches msi_processor’s AuxiliaryDataFile.data_ptr mappings (zarr v2):

nuc.zarr /gain/<band>, /offset/<band> float32 (detector,) PRNU correction g_d, o_d dark.zarr /dark_offset/<band> float32 scalar per-band dark k radiometric.zarr /gain/<band>, /offset/<band> float32 scalar DN→radiance G, O spectral.zarr /esun/<band> float32 scalar ESUN (toa reflectance) noise.zarr /alpha/<band>, /beta/<band> float32 scalar σ=√(α²+β·DN) (E2ES-side)

Convention. The NUC gain/offset follow the processor’s own two-point form (msi_processor.computing.radiometric.core.estimate_nuc): from a synthetic dark frame D and a diffuser flat F (both (line, detector)), g_d = (μ_F−μ_D)/(F̄_d−D̄_d), o_d = μ_F − g_d·F̄_d, plus the per-band dark k = μ_D that apply_nuc subtracts. The absolute radiometric.gain is derived from the diffuser (l_diff/(μ_F−μ_D) ≈ 1/cal_gain, offset = 0) and is self-consistent with the NUC, so the processor’s apply_nuc then dn_to_radiance recover A·L then L. The coefficients are derived (not the truth ADF) — the round-trip is therefore non-tautological (inverse-crime cure).

Scalars are stored as 0-d arrays (float(...)-able, as the processor reads them). The cal-DB is per-band (one across-track (detector,) vector per band), matching the processor’s per-band ADF.

class s2_msi_raw_generator.adf_writer.BandCal(band, nuc_gain, nuc_offset, dark_offset, radio_gain, radio_offset=0.0, esun=0.0, noise_alpha=0.0, noise_beta=0.0)#

Bases: object

Per-band calibration coefficients in the msi-processor ADF convention.

band: str#

dark_offset: float#

esun: float = 0.0#

noise_alpha: float = 0.0#

noise_beta: float = 0.0#

nuc_gain: ndarray#

nuc_offset: ndarray#

radio_gain: float#

radio_offset: float = 0.0#

s2_msi_raw_generator.adf_writer.nuc_two_point(dark_frame, flat_frame)#

Per-detector NUC (gain, offset, dark_offset) in the processor’s two-point convention.

Reproduces msi_processor.computing.radiometric.core.estimate_nuc exactly (dark + flat → g_d = (μ_F−μ_D)/(F̄_d−D̄_d), o_d = μ_F − g_d·F̄_d) plus the per-band dark k = μ_D that apply_nuc subtracts. dark_frame / flat_frame are (line, detector). Degenerate detectors (F̄_d = D̄_d) yield a non-finite gain — left as-is; the processor flags them via detect_bad_pixels.

Return type:: tuple[ndarray, ndarray, float]

s2_msi_raw_generator.adf_writer.write_calibration_db(out_dir, cals, *, unit='S2A', source='derived (CSM diffuser + dark calibration)', include_spectral=True, include_noise=True)#

Write the EOPF zarr ADF set (nuc / dark / radiometric [+ spectral + noise]).

The raw calibration acquisitions behind these coefficients are not ADF side-files: the pipeline’s calibration mode packages them as downlink Synthetic L0 products (S02MSIDCA dark, S02MSISCA sun-diffuser) — see the ICD.

Returns the list of written .zarr paths. A PROVENANCE.md is written beside them.

Return type:: list[Path]

s2_msi_raw_generator.caldb module#

Build the calibration database (EOPF ADF set) for the downstream L1PP processor.

Derives the radiometric calibration coefficients for all 13 bands via the two-reference (CSM sun-diffuser + dark) inverse-crime cure — in the msi-processor NUC convention — and writes them as EOPF zarr ADFs (nuc / dark / radiometric [+ noise]) with a PROVENANCE.md. This is Option Y of the generator ⇄ processor coupling: the generator produces the ADF; the processor keeps calibration internal. See s2_msi_raw_generator.adf_writer.

Driven by the build-caldb phase of scripts/run_pipeline.py.

s2_msi_raw_generator.caldb.build(out_dir, *, unit='S2A', n_det=400, seed=0, include_noise=True)#

Derive all 13 bands and write the cal-DB to out_dir. Returns the written ADF paths.

Return type:: list[Path]

s2_msi_raw_generator.caldb.derive_band_cal(band_name, *, unit='S2A', n_det=400, n_frames=256, seed=0)#

Derive one band’s cal-DB coefficients from synthetic dark + diffuser acquisitions.

Impresses the truth ADF to synthesise a dark and a sun-diffuser (flat) L0 acquisition, then derives the NUC (gain, offset) in the processor’s two-point convention plus the per-band dark and the absolute radiometric.gain = 1/cal_gain. Coefficients are derived, not the truth.

Return type:: BandCal

s2_msi_raw_generator.caldb.derive_from_acquisitions(band_name, dark, flat, *, l_diff=None, unit='S2A')#

Derive one band’s cal-DB coefficients from given dark + flat-field acquisitions.

The two acquisitions may come from derive_band_cal()’s internal synthesis or, in the pipeline’s calibration mode, from the packaged calibration-campaign Synthetic L0 products (dark S02MSIDCA, sun-diffuser S02MSISCA) — same numbers either way.

Return type:: BandCal

s2_msi_raw_generator.calibration module#

S2 in-flight radiometric calibration sub-set — synthetic CSM sun-diffuser + dark → derived ADF.

A two-reference radiometric calibration in Sentinel-2’s reflective domain: the high-signal reference is the on-board CSM sun-diffuser (a full-field, full-pupil Lambertian diffuser giving a near-uniform bright image), and the zero reference is a dark acquisition (CSM shutter closed / nighttime ocean). Per the public L1 ATBD §4.1.1.2.2:

D(j) = ⟨X_dark(i, j)⟩_i (dark from a dark acquisition) g(j) = A · ⟨L_diff⟩_i / ⟨X_diff(i, j) − D(j)⟩_i (relative response from the diffuser) with ⟨g(j)⟩_j = 1 ⇒ fixes the absolute calibration coefficient A.

This module (a) generates the synthetic dark + diffuser L0 acquisitions by impressing the true ADF through the reverse chain, then (b) derives D, g and A back from them — the estimated calibration a downstream processor would actually use, instead of the truth ADF. Closing this loop (impress truth → estimate → use the estimate) is the E2ES inverse-crime cure: residuals then reflect calibration uncertainty, not a tautology.

s2_msi_raw_generator.calibration.DIFFUSER_LEVEL_FACTOR = 1.5#: Diffuser radiance level as a multiple of Lref — bright but clip-free for every band (worst case B08 stays ≈2510 DN pre-quantization; S14 clips silently above 4095).

class s2_msi_raw_generator.calibration.DerivedCalibration(band, dark, relative_response, abs_coeff, l_diff)#

Bases: object

Calibration coefficients estimated from the synthetic diffuser + dark acquisitions.

abs_coeff: float#

band: str#

dark: ndarray#

l_diff: float#

relative_response: ndarray#

s2_msi_raw_generator.calibration.calibrate(adf, l_diff=None, *, n_dark=256, n_diffuser=256, seed=0)#

Run the full S2 calibration sub-set on the true adf: synthesise the dark + diffuser acquisitions, then derive the estimated D, g, A.

l_diff defaults to a bright diffuser radiance (≈1.5·Lref), staying within the dynamic range.

Return type:: DerivedCalibration

s2_msi_raw_generator.calibration.derive_dark(dark_acq)#

Estimate the per-detector dark D(j) = ⟨X_dark⟩_i (average over lines reduces noise).

Return type:: ndarray

s2_msi_raw_generator.calibration.derive_relative_response(diffuser_acq, dark_est, l_diff)#

Estimate the relative response g(j) (⟨g⟩ = 1) and the absolute coefficient A.

g(j) = A·L_diff / ⟨X_diff(i,j) − D(j)⟩_i then normalised so ⟨g(j)⟩_j = 1 (L1 ATBD).

Return type:: tuple[ndarray, float]

s2_msi_raw_generator.calibration.estimated_adf(adf, cal)#

Build a new BandADF using the estimated dark + relative response (not the truth).

PSF and the noise model stay as-is; only the per-detector dark/PRNU are replaced by the diffuser/dark-derived estimates — the coefficients a processor would actually apply.

Return type:: BandADF

s2_msi_raw_generator.calibration.synth_dark_acquisition(adf, n_lines, rng)#

Synthetic dark L0 acquisition (zero scene radiance → dark pedestal + sensor noise).

Return type:: ndarray

s2_msi_raw_generator.calibration.synth_diffuser_acquisition(adf, l_diff, n_lines, rng)#

Synthetic sun-diffuser L0 acquisition (uniform full-field radiance l_diff → raw DN).

The diffuser is a flat field, so the radiance frame is uniform; the reverse chain impresses the true relative response (PRNU), dark and noise — exactly what the on-board CSM diffuser measures.

Return type:: ndarray

s2_msi_raw_generator.ccsds122 module#

CCSDS 122.0-B image-data compression — lossless subset (integer DWT + bit-plane coder).

Sentinel-2 compresses MSI video data onboard with the proprietary MRCPB wavelet scheme (bit-plane coding, “similar to JPEG 2000”); a CCSDS-compression ASIC is the documented alternative option (eoPortal, S2 CoReCi). This module implements that alternative — CCSDS 122.0-B — in its lossless profile, pure numpy, so the generator’s L0 ISP payloads carry genuinely wavelet-compressed image data and the ground segment (L1A side) can restore the exact DN.

Implemented per the Blue Book structure#

Integer DWT 9/7-M (CCSDS 122.0-B §3.3): three 2-D decomposition levels, lifting form with the specified floor(·+1/2) rounding and whole-sample symmetric extension; exactly invertible on integers.
Block/family structure (§4.1): one 8×8 pixel area → 1 DC + 63 AC coefficients in three families (parent, 2×2 children, 4×4 grandchildren per family), raster scan; gaggles of 16 blocks; segments of segment_blocks blocks.
Segment headers (§4.2): the Part-1A/3/4 semantic content (start/end flags, segment count, BitDepthDC/BitDepthAC, segment size, DWT type, pixel bit depth, image dimensions) is carried in an explicit little-endian layout (see parse_segment_headers); the stream is fully self-describing.
DC coefficient coding (§4.3): per-segment DPCM against a raw reference sample, zigzag signed→nonnegative mapping, per-gaggle Rice coding with brute-force parameter selection and an uncoded escape option.
Per-block BitDepthAC (§4.4): same DPCM + per-gaggle Rice machinery.

Documented divergences (kept deliberately small, listed in the ICD/ATBD)#

§4.5.3 word mapping / variable-length codes are not implemented. AC bit-plane coding keeps the stage semantics and scan order (significance → sign → refinement, plane-sequential from BitDepthAC−1 down to 0, block-major within a segment) but the per-stage bits are packed raw (numpy.packbits) instead of entropy-coded words. Consequence: bit-exact lossless with the full segment/gaggle/block/plane structure, but the stream is not interoperable with reference CCSDS-122 decoders (the matching decoder lives in this module) and ratios run below a full BPE.
Header field packing uses explicit byte-aligned little-endian fields (with section byte-lengths) rather than the Blue Book bit layout; the content mirrors Parts 1A/3/4.
Sections are byte-aligned; the Blue Book packs them contiguously.

The API is frame-oriented: compress_frame() → (payload bytes, CompressionStats) and decompress_frame() → the exact uint16 DN frame. Segments default to one block row (8 image lines) so packetization downstream maps segments to CCSDS space-packet groups with line-accurate datation.

class s2_msi_raw_generator.ccsds122.CompressionStats(raw_bytes, compressed_bytes, n_segments, n_blocks, pixel_bit_depth, dc_bytes, bitdepth_ac_bytes, ac_bytes, header_bytes)#

Bases: object

Byte accounting of one compressed frame.

ac_bytes: int#

bitdepth_ac_bytes: int#

property bits_per_pixel: float#: Compressed bits per pixel.

compressed_bytes: int#

dc_bytes: int#

header_bytes: int#

n_blocks: int#

n_segments: int#

pixel_bit_depth: int#

property ratio: float#: Raw ÷ compressed size (packed-sample raw, pixel_bit_depth bits/px).

raw_bytes: int#

s2_msi_raw_generator.ccsds122.GAGGLE_BLOCKS = 16#: Blocks per gaggle (CCSDS 122.0-B §4.1).

s2_msi_raw_generator.ccsds122.MAGIC = b'C122LSv1'#: Frame-stream magic (8 bytes) + container version.

s2_msi_raw_generator.ccsds122.compress_frame(dn, *, pixel_bit_depth=12, segment_blocks=None)#

Losslessly compress a 2-D DN frame → (payload, stats).

segment_blocks defaults to one block row (width // 8), aligning each segment with 8 image lines. The payload is fully self-describing (see module docstring).

Return type:: tuple[bytes, CompressionStats]

s2_msi_raw_generator.ccsds122.decompress_frame(payload)#

Exact inverse of compress_frame() → uint16 frame.

Return type:: ndarray

s2_msi_raw_generator.ccsds122.dwt97m_forward(frame, levels=3)#

3-level (default) 2-D integer 9/7-M DWT → subband dict LL<n>, HL<l>, LH<l>, HH<l>.

Rows are transformed first, then columns; the recursion descends into LL. Input dimensions must be divisible by 2**levels.

Return type:: dict[str, ndarray]

s2_msi_raw_generator.ccsds122.dwt97m_inverse(bands, levels=3)#

Exact inverse of dwt97m_forward().

Return type:: ndarray

s2_msi_raw_generator.ccsds122.parse_segment_headers(payload)#

Header inventory of a compressed stream (frame fields + per-segment Part-1A content).

Return type:: dict

s2_msi_raw_generator.ccsds122.segment_byte_bounds(payload)#

Byte offsets of packetization groups: [0, seg1_off, …].

The frame header is folded into the first group, so joining the groups reproduces the exact stream — the property decompress_frame() needs after packet reassembly.

Return type:: list[int]

s2_msi_raw_generator.datation module#

GPS/OBT line datation for the L0 ISP — OBT/GPS/TAI line timing (ADF_DATAT model).

Replaces the placeholder t0 = 0 CUC time with a Sentinel-2 on-board time derived from a acquisition epoch. The MSI line-datation model (ADF_DATAT) is

t(line, band) = epoch + line · line_period + time_shift(band)

expressed here on the GPS timescale (the S2 on-board clock is GPS-derived; the 32-bit CUC coarse field holds the GPS second-of-epoch). UTC and TAI are offered for the product metadata (orbit_ephemeris carries TAI/UTC/UT1). msi-processor does not consume the CUC time for L1B, but EOPF EOQC’s Datation_Sync / Time_Correlation checks read the datation metadata.

class s2_msi_raw_generator.datation.Datation(epoch_utc='2024-04-03T10:24:15Z', line_period_s=0.0015658736, time_shift_s=<factory>)#

Bases: object

Line-datation model for one acquisition (the ADF_DATAT parameters we need for the Synthetic L0).

band_time_stamp()#

Per-band first-line GPS time stamp for the Synthetic L0 other_metadata (band number → {unit, value}).

Return type:: dict[str, dict]

property epoch: datetime#

epoch_utc: str = '2024-04-03T10:24:15Z'#

property gps_epoch_s: float#: GPS second-of-epoch of the first line (the CUC/OBT t0).

line_period_s: float = 0.0015658736#

line_time_gps(line, band=None)#

GPS seconds of line (optionally with the per-band along-track time_shift).

Return type:: float

line_time_utc(line, band=None)#

Return type:: datetime

span_utc(n_lines)#

(start, end) UTC ISO for the acquisition (first → last line).

Return type:: tuple[str, str]

time_shift_s: dict[str, float]#

s2_msi_raw_generator.datation.utc_to_gps_seconds(utc)#

Seconds on the GPS timescale since the GPS epoch (1980-01-06T00:00:00).

Return type:: float

s2_msi_raw_generator.env module#

Environment configuration — single source of truth from .env.

All pipeline scripts load the repository .env at startup. Required path variables must be set; there are no silent fallbacks to ~/data-store or baked-in defaults.

s2_msi_raw_generator.env.adf_eopf_dir()#

Return type:: Path

s2_msi_raw_generator.env.aux_dir()#

Return type:: Path

s2_msi_raw_generator.env.ensure_repo_on_path()#

Allow python scripts/run_pipeline.py without a prior pip install -e.

Return type:: None

s2_msi_raw_generator.env.env_int(name, default)#

Return type:: int

s2_msi_raw_generator.env.find_adf_eopf(adf_type)#

First S0*_ADF_<adf_type>_*.json under {S2_AUX_DIR}/adf-eopf.

Return type:: str | None

s2_msi_raw_generator.env.find_framing_table()#

First framing_lines_*.json under {S2_AUX_DIR}/framing.

Return type:: Path | None

s2_msi_raw_generator.env.framing_dir()#

Return type:: Path

s2_msi_raw_generator.env.gipp_dir()#

Return type:: Path

s2_msi_raw_generator.env.init_env(*, require=False)#

Load .env and optionally enforce required path variables.

Return type:: None

s2_msi_raw_generator.env.l0_input()#

Return type:: Path

s2_msi_raw_generator.env.l1b_input()#

Return type:: Path

s2_msi_raw_generator.env.load_dotenv(path=None)#

Load KEY=VALUE lines from .env into os.environ (does not override existing).

Return type:: None

s2_msi_raw_generator.env.output_dir()#

Return type:: Path

s2_msi_raw_generator.env.require_paths()#

Exit if any mandatory path variable is unset.

Return type:: None

s2_msi_raw_generator.env.store_paths()#

Pipeline store layout under OUTPUT_DIR.

Return type:: dict[str, Path]

s2_msi_raw_generator.forward_radiometric_atbd module#

Original implementation of the Sentinel-2 on-ground radiometric model (public L1 ATBD §4.1.1).

Given the per-pixel GIPP coefficients (s2_msi_raw_generator.gipp.DetectorEq), this provides the forward radiometric correction (raw L1A → corrected L1B) and its exact inverse (the reverse-E2ES impress step). The model is the published one:

Z = X − D                                   (dark-signal subtraction; D = COEFF_D)
Y = G(Z)   with   G = A·Z³+B·Z²+C·Z         (VNIR cubic relative-response)
            or    A1·Z  | A2·Z+(A1−A2)·Zs   (SWIR bilinear, knee at Z = Zs)

Everything is written from the public ATBD equations; no processor source is used. Coefficients are per across-track pixel (act) and broadcast over image lines.

s2_msi_raw_generator.forward_radiometric_atbd.column_fpn(frame, smooth=31)#

High-frequency across-track fixed-pattern-noise metric: std of (column means − running mean).

A PRNU-striped image has higher column FPN before equalization than after.

Return type:: float

s2_msi_raw_generator.forward_radiometric_atbd.forward_correct(x_raw, eq)#

Forward radiometric correction L1A→L1B: Y = G(X − D) (subtract dark, equalize).

Return type:: ndarray

s2_msi_raw_generator.forward_radiometric_atbd.forward_equalize(z, eq)#

Apply the relative-response model Y = G(Z) (dark-subtracted signal Z → equalized Y).

Return type:: ndarray

s2_msi_raw_generator.forward_radiometric_atbd.inverse_equalize(y, eq, iters=6)#

Invert G: equalized Y → dark-subtracted signal Z (impress the per-pixel relative response).

Return type:: ndarray

s2_msi_raw_generator.forward_radiometric_atbd.reapply_onboard_eq(z_signal, reob2)#

S12 — re-apply the on-board bilinear equalization non-linearity (undo forward step 1).

The forward inverse_equalization un-does the on-board companding Z = Y/a1 | (Y−(a1−a2)zs)/a2 (X = Z + d); its E2ES re-application on the dark-subtracted signal z (downlink- consistent) is y = a1·z for z ≤ zs else a2·z + (a1−a2)·zs (REOB2 a1,a2,zs). The REOB2 dark d (raw-detector domain, ≈455 DN) cancels the raw-detector dark and is not re-added here — the downlink L0 pedestal is the separate l0_dark_level term. For S2B a1≈1.005, a2≈0.995 so this is a sub-percent gain with a knee at z ≈ zs.

Return type:: ndarray

s2_msi_raw_generator.forward_radiometric_atbd.restage_swir_lines(img, shifts, kernel=None, method='shift')#

S8 — re-introduce the SWIR staggered readout (reverse of the forward re-arrangement).

shifts[c] ∈ {−1, 0, +1} flags which across-track columns move and the direction. method "shift" (B11/B12) rolls the flagged column by ±1 whole line; "interp" (B10) applies the ±1/3-line sub-pixel shift as a 3-tap kernel convolution (kernel for +, reversed for −; lossy — the forward convolution is not exactly invertible). Along-track roll commutes with the per-column radiometric ops, so the exact placement in the chain is not critical. shifts is centre-aligned to the frame width (SWIR blind columns sit at the edges).

Return type:: ndarray

s2_msi_raw_generator.forward_radiometric_atbd.reverse_impress(y, eq)#

Reverse (E2ES) of forward_correct(): corrected L1B → raw L1A X = G⁻¹(Y) + D.

Return type:: ndarray

s2_msi_raw_generator.forward_radiometric_atbd.reverse_l1b_to_l0(l1b_dn, eq, *, radio_offset_l1b, l0_dark_level, unbin_factor=1, dn_max=4095, swir_shift=None, defective_cols=None, onboard_eq=None, nodata=0.0)#

E2ES reverse of the L0→L1B radiometric chain: S2 L1B digital counts → Synthetic L0 raw uint16 DN.

The SentiWiki / EOPF forward chain (ON steps) is on-board-eq⁻¹(REOB2) → dark → blind → crosstalk(RCRCO) → relative-response(REQOG) → SWIR-rearr(RSWIR) → defective(RDEPI) → restoration[OFF] → binning → +RADIO_ADD_OFFSET. This inverts it in the downlink DN domain (where both S2 L1B and L0 live), undoing the steps in reverse order:

z    = G⁻¹(L1B + radio_offset_l1b)          # undo offset (S4) + impress relative response (S7)
z    = a·z (on-board-eq non-linearity)      # S12 — undo REOB2 (a1≈a2≈1, sub-percent; optional)
raw  = z + l0_dark_level · (D/⟨D⟩)           # S11 — L0-domain dark pedestal; DSNU shape from COEFF_D
raw  = repeat(raw, ×unbin_factor)           # S5 — un-bin (60 m); replication (sub-pixel irrecoverable)
raw  = restage_swir_lines(raw, swir_shift)  # S8 — re-introduce the staggered SWIR readout
raw[:, defective_cols] = nodata             # S10 — re-stamp the RDEPI defective columns

Dark is the downlink-domain l0_dark_level (≈50 DN), not the raw-detector eq.dark (COEFF_D ≈440 DN, a different domain — its ≈455 DN cancels the REOB2 d, so REOB2 contributes only its a1/a2 non-linearity here); eq.dark supplies only the unitless DSNU column shape.

Skipped — deliberately: MTF restoration/deconvolution and de-noising (forward step 8) are feature_flag_with_deconvolution/denoising = False in the operational payload (SentiWiki: “restoration disabled by default — instrument MTF already high”), so the L1B still carries the full instrument PSF and its noise realisation; re-blurring (S6 PSF) or re-noising (S13) would double-count and is not applied. Crosstalk (S9, RCRCO) is applied at the phase level (needs the neighbouring same-resolution bands; ≈0 for S2A/B).

Optional full-chain inputs (all None → the validated radiometric-only reverse): swir_shift = (shifts, kernel, method) from read_rswir_eopf(); defective_cols = RDEPI singularity_columns; onboard_eq = REOB2 coefficients from read_reob2_eopf(). Validated against the 2024-04-08 S2B PPB pair (all 13 bands): median ≤~5 %, active-region column FPN matches, CCSDS-122/ISP round-trip bit-exact; S8 brings the SWIR (B11/B12) images into spatial agreement.

Return type:: ndarray

s2_msi_raw_generator.gipp module#

Reader for Sentinel-2 operational GIPP calibration data.

Production layout: band-organised JSON under S2_GIPP_DIR (aux/gipp-json/{Bxx}/S2B_ADF_*.json), each file wrapping the documented GS2_* schema (XML-to-JSON conversion). Unit tests may still use flat XML fixtures (*GIP_*.xml).

The parser turns GIPP into per-band / per-detector / per-pixel numpy arrays the reverse chain needs:

R2EQOG — the on-ground equalization file: per-pixel dark signal D and the relative-response (PRNU) gains — VNIR cubic (A, B, C) (Y = A·Z³+B·Z²+C·Z) or SWIR bilinear (A1, A2, Zs) (Y = A1·Z for Z≤Zs else A2·Z+(A1−A2)·Zs). Per the public Sentinel-2 L1 ATBD §4.1.1, C (VNIR) / A1 (SWIR) is the dominant per-pixel relative gain.
R2DEPI — defective (saturated + blind) pixel columns per band/detector.
BLINDP — blind-pixel column lists per band/detector/side.
R2PARA — per-band equalization/offset flags + radiometric offsets (−100 L1B / −1000 L1C).
R2CRCO / R2BINN — crosstalk matrix (≈0 for S2A) / 60 m binning kernel.

Band order follows sensor.BANDS (band_id 0…12 → B01…B08, B8A, B09…B12).

class s2_msi_raw_generator.gipp.BandEq(band, tdi, detectors, source='GIPP R2EQOG')#

Bases: object

band: str#

detectors: dict[int, DetectorEq]#

property npix: int#

source: str = 'GIPP R2EQOG'#

tdi: bool#

class s2_msi_raw_generator.gipp.DetectorEq(model, dark, coeffs)#

Bases: object

Per-detector R2EQOG coefficients (each array is per across-track pixel, length NB_OF_PIXELS).

coeffs: dict[str, ndarray]#

dark: ndarray#

model: str#

property rel_gain: ndarray#

C for VNIR, A1 for SWIR.

Type:: Dominant per-pixel relative-response (PRNU) gain

class s2_msi_raw_generator.gipp.GippSet(gipp_dir, equalization=<factory>, defective=<factory>, blind=<factory>, params=None, crosstalk=<factory>, crosstalk_matrix=None, rswir_adf=None, reob2_adf=None)#

Bases: object

All GIPP calibration data needed by the reverse chain, parsed from gipp_dir.

The full-reverse extras (rswir_adf = SWIR shift map, reob2_adf = on-board equalization) are EOPF ADF json paths read lazily through swir_shift() / onboard_eq() — those ADFs are large (REOB2 ≈ 125 MB) and only a handful of (band, detector) pairs are ever needed.

band(name)#

Return type:: BandEq

blind: dict[str, dict[int, ndarray]]#

crosstalk: dict[str, ndarray]#

crosstalk_matrix: ndarray | None = None#

defective: dict[str, dict[int, ndarray]]#

equalization: dict[str, BandEq]#

gipp_dir: str#

onboard_eq(band, detector)#

On-board-equalization coefficients (REOB2) for a (band, detector), or None.

Return type:: dict[str, ndarray] | None

params: RadioParams | None = None#

reob2_adf: str | None = None#

rswir_adf: str | None = None#

swir_shift(band, detector)#

SWIR staggered-readout shift map for a (band, detector), or None if unavailable.

Return type:: tuple[ndarray, ndarray, str] | None

class s2_msi_raw_generator.gipp.RadioParams(radiance_offset_l1b, reflectance_offset_l1c, equalization_flag, offset_flag, dsnu_flag)#

Bases: object

dsnu_flag: dict[str, bool]#

equalization_flag: dict[str, bool]#

offset_flag: dict[str, bool]#

radiance_offset_l1b: dict[str, int]#

reflectance_offset_l1c: dict[str, int]#

s2_msi_raw_generator.gipp.load_gipp_set(gipp_dir, bands=('B01', 'B02', 'B03', 'B04', 'B05', 'B06', 'B07', 'B08', 'B8A', 'B09', 'B10', 'B11', 'B12'), eqog_adf=None, rswir_adf=None, reob2_adf=None, rcrco_adf=None)#

Load the full GIPP set from gipp_dir (JSON gipp-json or XML test fixtures).

Equalization comes from gipp_dir (ADF_REQOG per band). Optional eqog_adf forces EOPF data_vars REQOG (V&V only — not used by the production pipeline). rswir_adf / reob2_adf / rcrco_adf enable the full reverse chain (S8/S9/S12) from adf-eopf.

Return type:: GippSet

s2_msi_raw_generator.gipp.parse_eqog_adf_epoch(adf_path)#

Parse EOPF ADF filename (S2A_ADF_REQOG_<start>_<stop>_<creation>.json, or the PSFD platform spelling S02B_ADF_…) → its applicability-start / validity-stop / creation epochs (UTC ISO). Empty dict if it doesn’t match. platform is normalized to S2x.

Return type:: dict

s2_msi_raw_generator.gipp.read_blindp(gipp_dir)#

Blind-pixel 0-based column indices (valid ∪ non-valid, both sides), per band → detector.

Return type:: dict[str, dict[int, ndarray]]

s2_msi_raw_generator.gipp.read_blindp_json(gipp_dir)#

Return type:: dict[str, dict[int, ndarray]]

s2_msi_raw_generator.gipp.read_r2crco(gipp_dir)#

Per-band crosstalk row (OPTICAL+ELECTRICAL summed). ≈0 for S2A.

Return type:: dict[str, ndarray]

s2_msi_raw_generator.gipp.read_r2crco_json(gipp_dir)#

Return type:: dict[str, ndarray]

s2_msi_raw_generator.gipp.read_r2depi(gipp_dir)#

Defective (saturated ∪ blind) 0-based across-track column indices, per band → detector.

Return type:: dict[str, dict[int, ndarray]]

s2_msi_raw_generator.gipp.read_r2depi_json(gipp_dir)#

Return type:: dict[str, dict[int, ndarray]]

s2_msi_raw_generator.gipp.read_r2eqog_band(gipp_dir, band)#

Parse per-band equalization (JSON gipp-json or XML fixture layout).

Return type:: BandEq

s2_msi_raw_generator.gipp.read_r2eqog_eopf(adf_json_path, band)#

Parse EOPF ADF_REQOG (S2[AB]_ADF_REQOG_…json) → the same BandEq as the XML read_r2eqog_band().

The EOPF ADF stores, per band <b> (lower-case; b8a for B8A) over 12 detectors: VNIR cubic <b>/coeff_{a,b,c} (→ A,B,C) or SWIR bilinear <b>/coeff_{a1,a2,zs} (→ A1,A2,Zs), plus <b>/coeff_d = dark signal. coeff_d may carry along-track sub-lines ((det, n_line, act)); these are collapsed to their mean, matching the XML COEFF_D convention in _parse_coeff_d().

Return type:: BandEq

s2_msi_raw_generator.gipp.read_r2eqog_json(gipp_dir, band)#

Parse per-band ADF_REQOG JSON → BandEq.

Return type:: BandEq

s2_msi_raw_generator.gipp.read_r2para(gipp_dir)#

Return type:: RadioParams

s2_msi_raw_generator.gipp.read_r2para_json(gipp_dir)#

Return type:: RadioParams

s2_msi_raw_generator.gipp.read_rcrco_eopf(adf_json_path)#

EOPF ADF_RCRCO → the combined crosstalk matrix dtalk[k, l] (optical + electrical).

13×13 over band_k (impacted) × band_l (impacting), in sensor.BANDS order. The forward correction is X_corr_k = X_k − Σ_l dtalk[k,l]·X_l; the E2ES re-application adds it back. For S2A/B optical is exactly 0 and electrical peaks at ≈0.004 — a near-no-op kept for completeness.

Return type:: ndarray

s2_msi_raw_generator.gipp.read_reob2_eopf(adf_json_path, band, detector, table='new_table')#

EOPF ADF_REOB2 → per-pixel on-board-equalization coefficients for one band/detector.

The forward L1B step 1 (inverse_equalization) undoes the on-board bilinear equalization:

Z = where(Y ≤ a1·zs, Y/a1, (Y − (a1−a2)·zs)/a2);   X = Z + d

so the E2ES re-application (raw X → downlink Y) is Y = where(Z ≤ zs, a1·Z, a2·Z + (a1−a2)·zs) with Z = X − d (see reapply_onboard_eq()). Returns {a1, a2, zs, d} each (act,) from <band>/<table>/coeff_* (default the operational new_table). For S2B a1≈1.005, a2≈0.995 (near-unity slope) and d≈455 (the raw-detector dark, i.e. the R2EQOG COEFF_D domain).

Return type:: dict[str, ndarray]

s2_msi_raw_generator.gipp.read_rswir_eopf(adf_json_path, band, detector)#

EOPF ADF_RSWIR → the SWIR staggered-readout shift map for one band/detector.

Returns (shifts, kernel, method):

shifts — per across-track column integer flag in {−1, 0, +1} (swir_band_list/ swir_band/detector at [band, detector, :]). The sign marks which columns move and the direction; the magnitude is set by method (whole line vs sub-pixel).
kernel — the interpolation_filter/coefs 3-tap sub-pixel filter (only used for B10).
method — "shift" (B11/B12: ±1 whole-line roll) or "interp" (B10: ±1/3 line via kernel convolution).

Only B10/B11/B12 have an entry; band must be one of those (else KeyError).

Return type:: tuple[ndarray, ndarray, str]

s2_msi_raw_generator.gipp.temporal_validity(adf_epoch, acquisition_utc, warn_years=1.0)#

Check an ADF’s temporal applicability to an acquisition.

within_validity uses the ADF’s declared [applicability_start, valid_stop] window — but note the 2100 stop is an open-ended placeholder, not proof of applicability. So the operational signal is gap_years = |acquisition − applicability_start|: the satellite’s radiometric state drifts (monthly-refreshed R2EQOG/R2ABCA), and a stale ADF is flagged (warn) when the gap exceeds warn_years or the acquisition falls outside the declared window.

Return type:: dict

s2_msi_raw_generator.import_l0 module#

Import a public distribution L0 image product as a PDI-style L1A input.

The bridge does not synthesize data: it copies one public detector’s per-band DN arrays into the measurements/DD01/Bxx/l1a_raw_image layout consumed by the pipeline, preserving acquisition identity in root STAC metadata and recording per-band provenance.

s2_msi_raw_generator.import_l0.convert(public_l0_path, out_dir, *, detector=1, bands=None, jobs=1)#

Convert one public L0 detector into a PDI-style L1A Zarr directory.

jobs is accepted for pipeline symmetry; writes stay serial so memory remains bounded by one band array at a time.

Return type:: dict

s2_msi_raw_generator.import_l0.read_public_l0_identity(public_l0_path)#

Read acquisition identity from a public L0 Zarr/Zip product.

Return type:: dict

s2_msi_raw_generator.import_l0.write_l1a_product(out_path, frames, *, datation=None, platform='Sentinel-2B', orbit=None, footprint=None, source_l1b=None, provenance=None)#

Write reconstructed raw-DN frames as EOPF L1A product (multi-detector).

This is the reverse chain’s materialised intermediate (reverse-l1b phase): the S2 L1B → Synthetic L0 reconstruction stops at the raw-count L1A domain and persists it before the separate L1A→Synthetic L0 packaging (CCSDS/ISP) step. frames maps (detector 1–12, band "B03") to a uint16 (line, column) raw-DN array; each is written to measurements/DD{dd}/{BAND}/l1a_raw_image (the layout read_l1a_raw() consumes) and re-read to assert a bit-identical round-trip. Writes stay serial (one array at a time) — parallel writes corrupt the measurements arrays on the NFS-backed data-store.

Root attrs carry an L1A STAC identity (processing:level=L1A, product:type=S02MSIL1A) plus the reverse provenance. Returns a summary dict (output path, per-frame shape/crc32).

Return type:: dict

s2_msi_raw_generator.inventory module#

Data-store inventory and consistency reporting.

The scanner is intentionally metadata-only: it walks the store layout, reads Zarr attributes from directories or .zarr.zip archives with stdlib JSON/zipfile support, and classifies products without loading image arrays.

s2_msi_raw_generator.inventory.assign_groups(records)#

Annotate records with acquisition grouping and comparability hints.

Return type:: None

s2_msi_raw_generator.inventory.consistency_findings(records, store)#

Build the consistency finding list described in the plan.

Return type:: list[dict]

s2_msi_raw_generator.inventory.parse_name_lenient(name)#

Parse PSFD names first, then known public/PSD-like names.

Return type:: dict

s2_msi_raw_generator.inventory.read_zarr_identity(path)#

Read best-effort acquisition identity from a Zarr directory or zip archive.

Return type:: dict

s2_msi_raw_generator.inventory.render_markdown(records, findings)#

Return type:: str

s2_msi_raw_generator.inventory.scan_store(store)#

Return inventory records for the known data-store layout.

Return type:: list[dict]

s2_msi_raw_generator.inventory.write_outputs(store)#

Scan store and write INVENTORY.md plus report/inventory.json.

Return type:: dict

s2_msi_raw_generator.io module#

Lightweight reader for Sentinel-2 EOPF L1A/L1B Zarr products (no full eopf dependency).

Reads the per-detector / per-band radiance arrays straight from the Zarr store with zarr. The full EOPF CPM (eopf==2.8.1, optional extra) is only needed to write spec-compliant Synthetic L0 EOProducts (Increment 2) and for the round-trip against the pinned msi-processor.

s2_msi_raw_generator.io.read_l1a_raw(path, detector, band, *, lines=None, dtype=<class 'numpy.float64'>)#

Read one detector/band raw-DN array from EOPF L1A product.

L1A uses measurements/DD{dd}/B{band}/l1a_raw_image (uppercase DDnn/Bxx, image name l1a_raw_image) — the raw instrument counts (with dark + PRNU still present), unlike the L1A/L1B radiance reader read_l1b_band() (lowercase dDD/bXX/img).

dtype defaults to float64 (the round-trip V&V convention); pass np.uint16 to read full frames without the 4× float64 memory spike, or None for the stored dtype.

Return type:: ndarray

s2_msi_raw_generator.io.read_l1b_band(path, detector, band, *, lines=None)#

Read one detector/band radiance array from an L1A/L1B product.

Parameters:

path (str)
detector (int)
band (str)
lines (slice | None)

Return type:

ndarray

s2_msi_raw_generator.isp module#

S15 — CCSDS Instrument Source Packet (ISP) generation + SAD telemetry.

Implements the ATBD §5.S15 step: package the Synthetic L0 detector/band frames into CCSDS Space Packets (CCSDS 133.0-B primary header + a CUC secondary time header), and generate Satellite Ancillary Data (SAD) packets per APID. Timestamps derive from sensor.LINE_PERIOD_MS.

Layout (ATBD Annex A.9):

measurements/d{DD}/b{BB}/{isp, isp_offsets, packet_data_length} — CCSDS space packets whose payloads carry the CCSDS-122-compressed image data (module ccsds122; ICD-IF-ISP).
conditions/anc_data/s{APID}/isp + packet_data_length — SAD/housekeeping telemetry.

s2_msi_raw_generator.isp.DEFAULT_MAX_PAYLOAD = 8192#: Default maximum packet-data-field payload (octets, excl. the CUC secondary header).

s2_msi_raw_generator.isp.apid_for(detector, band_index, base=1024)#

Deterministic 11-bit APID for a (detector, band) science stream.

Return type:: int

s2_msi_raw_generator.isp.build_primary_header(apid, seq_count, data_len, *, packet_type=0, sec_hdr_flag=1, seq_flags=3, version=0)#

CCSDS 133.0-B 6-octet primary header (big-endian).

data_len is the CCSDS Packet Data Length field = (octets in the packet data field) − 1.

Return type:: bytes

s2_msi_raw_generator.isp.build_sad_packets(apid, n_packets, *, t0_seconds=0.0, period_s, payload=b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00')#

Satellite Ancillary Data (housekeeping) packets for one APID.

Returns (isp, packet_data_length): isp is uint8 (n_packets, header+payload), packet_data_length is uint16 (n_packets,).

Return type:: tuple[ndarray, ndarray]

s2_msi_raw_generator.isp.cuc_time(t_seconds)#

6-octet CCSDS Unsegmented time Code: 4 octets coarse (s) + 2 octets fine (1/65536 s).

Return type:: bytes

s2_msi_raw_generator.isp.frame_isp_headers(frame, apid, *, t0_seconds=0.0, line_period_s)#

Legacy per-line ISP headers (superseded by packetize_stream() in the canonical Synthetic L0).

Returns (headers, packet_data_length):

headers — uint8 (n_lines, ISP_HEADER_LEN) primary + CUC-time header per line.
packet_data_length — uint16 (n_lines,) octet count of each packet’s data field (= CUC secondary header + one image line of uint16 samples).

Return type:: tuple[ndarray, ndarray]

s2_msi_raw_generator.isp.iter_packets(buf)#

Iterate CCSDS packets in a concatenated stream → (header dict, cuc_seconds, body bytes).

Shared by our reassembly and by structural scans of ISP files: the primary header’s Packet Data Length field walks the stream; a stream is well-formed iff packets tile it exactly. body excludes the CUC secondary header.

s2_msi_raw_generator.isp.packetize_stream(payload, apid, *, segment_bounds, segment_times_gps, max_payload=8192, seq_start=0)#

Carry a byte stream in CCSDS space packets, honouring segment grouping.

segment_bounds are the byte offsets of segment starts within payload (ascending, segment_bounds[0] == 0); each segment is sliced into packets of at most max_payload data octets and flagged SEQ_FIRST/SEQ_CONT/SEQ_LAST (SEQ_STANDALONE when a segment fits in one packet). Every packet carries a CUC secondary header stamped with its segment’s GPS time (segment_times_gps, seconds, one per segment). The 14-bit sequence counter starts at seq_start and is continuous across the whole stream.

Returns (isp, offsets, lengths): the concatenated packet stream (uint8 1-D), the byte offset of each packet within it (uint64), and each packet’s CCSDS Packet Data Length field value + 1 = data-field octet count (uint32).

Return type:: tuple[ndarray, ndarray, ndarray]

s2_msi_raw_generator.isp.parse_cuc_time(b)#

Inverse of cuc_time() — 6 octets → seconds (4-octet coarse + 2-octet fine / 65536).

Return type:: float

s2_msi_raw_generator.isp.parse_primary_header(b)#

Inverse of build_primary_header() (for tests / decode).

Return type:: dict

s2_msi_raw_generator.isp.reassemble_segments(buf)#

Inverse of packetize_stream(): packets → per-segment byte streams.

Enforces seq_flags grammar (FIRST → CONT* → LAST, or STANDALONE) and 14-bit sequence-counter continuity; raises ValueError on gaps or malformed flag sequences.

Return type:: list[bytes]

s2_msi_raw_generator.l0product module#

Assemble a synthetic Synthetic L0 RAW EOProduct (Increment 2, ICD-IF-Synthetic L0).

Writes the EOPF L0 Zarr structure (ATBD Annex A.9 / REQ-FUNC-030–036):

measurements/d{DD}/b{BB}/band{BB} uint16 (line, column) — 12 det × 13 bands = 156 arrays quality/d{DD}/b{BB}/mask uint8 — 156 masks root attrs: stac_discovery (geometry/bbox/orbit/datetime) + other_metadata (sensor-config, datation)

Uses zarr (v2 format for EOPF/msi-processor interoperability), not the full eopf CPM. Line timing / STAC datetime come from an Datation; when with_isp is set the CCSDS ISP headers + conditions/anc_data/... SAD telemetry (S15) are written.

s2_msi_raw_generator.l0product.build_root_metadata(*, platform='Sentinel-2A', datation=None, n_lines=1, active_detectors, footprint=None, orbit=None, eopf_type='S2MSIL0_', operation_mode='NOBS', datatake_type='INS-NOBS')#

Root STAC + sensor-configuration metadata (values; REQ-FUNC-033/034/035/038/048).

eopf_type/operation_mode/datatake_type identify the datatake kind (PSFD §3 vocabulary): nominal observation NOBS/INS-NOBS (default), dark calibration DASC/INS-DASC (S2MSIDCA), sun-diffuser calibration ABSR/INS-ABSR (S2MSISCA) — see ICD-IF-L0-CAL for the source notes.

Return type:: dict

s2_msi_raw_generator.l0product.decode_verify_band(canon_path, det, bname, l1a_path, line_slice)#

Ground-decode one band of a canonical Synthetic L0 and verify it against the original L1A DN.

Uses the CONSUMER decoder (msi-processor ground_decode) when importable, cross-checked bit-exactly against the E2ES reference read_l0_isp_dn(); falls back to the reference alone. Returns (reconstructed_dn, bit_exact_vs_original, cross_check) — cross_check is None without the consumer, else the decoder-agreement verdict. Module-level (picklable) so the driver’s ground-decode phase can fan bands out to a process pool (decompression is pure-Python/GIL-bound).

Return type:: tuple[ndarray, bool, bool | None]

s2_msi_raw_generator.l0product.read_l0_isp_dn(path, det, bname)#

Ground decompression: canonical Synthetic L0 ISP stream → the exact uint16 DN frame.

The S2 L1B chain L1A-side operation (SentiWiki: decompression happens at L1A): read measurements/d{DD}/b{BB}/isp, reassemble the packet groups (reassemble_segments() — seq_flags + continuity enforced), join them back into the CCSDS-122 stream and decode it bit-exactly.

Return type:: ndarray

s2_msi_raw_generator.l0product.reverse_to_l0_frames(l1b_frames, *, seed=0, adfs=None, unit='S2A')#

Run the MVP reverse chain on each (detector, band) radiance frame → uint16 Synthetic L0 DN.

Return type:: dict[tuple[int, str], ndarray]

s2_msi_raw_generator.l0product.write_l0_decoded_product(out_path, frames, *, datation=None, platform='Sentinel-2B', orbit=None, footprint=None, eopf_type='S2MSIL0_')#

Write decoded raw-count frames as EOPF L0 product format-identical to the reference ESA L0-zarr layout.

The archived EOPF L0 stores the losslessly-decompressed counts as measurements/d{DD}/b{BB}/img (uint16, dims alt,act) plus per-band decode-quality group attrs (decoding_error_number, valid_line_percentage …), verified against the S02MSIL0__…TC7D.zarr. This makes the Synthetic L0 format-identical to the ESA product for a direct array comparison (validate-l0); the compressed as-downlinked transport lives in the sibling L0plus (write_l0_product()). Frames must be uint16 (line, column).

Return type:: str

s2_msi_raw_generator.l0product.write_l0_opencontainer(out_path, band_frames, *, masks=None, datation=None, platform='Sentinel-2A', footprint=None, orbit=None, emit_qc=True)#

Write the open-container L0 that msi-processor’s l0_decode ingests (REQ-FUNC-042).

Layout (the decoded form the processor reads directly, distinct from the canonical 156-array product): measurements/detector/<BAND> uint16 (line, detector), quality/l0_flags/<BAND> uint16 (QAFlag seed), conditions/{time,orbit,attitude} per-line telemetry, + shared root metadata. The hard invariant for the downstream nuc ADF: nuc.gain[band] length must equal measurements/detector/<band>’s detector-axis width — so build the cal-DB at the same n_det.

Return type:: str

s2_msi_raw_generator.l0product.write_l0_product(out_path, frames, *, platform='Sentinel-2A', datation=None, datetime_iso=None, masks=None, footprint=None, orbit=None, emit_qc=True, with_isp=False, isp_max_payload=8192, store_decoded=True, eopf_type='S2MSIL0_', operation_mode='NOBS', datatake_type='INS-NOBS', jobs=1)#

Write the Synthetic L0 RAW EOProduct Zarr to out_path and return it.

Each frame must be uint16 in [0, DN_MAX]. A quality mask is written per frame (saturated pixels flagged where DN ≥ DN_MAX if no explicit mask is given). The datation (Datation) sets the GPS/OBT line timing and the STAC datetime span; footprint/orbit set the STAC geometry/orbit (S2A defaults). datetime_iso is a deprecated shortcut for Datation(epoch_utc=...).

When with_isp is set (S15) the band’s image data is CCSDS-122 lossless compressed (ccsds122) and carried as CCSDS space packets (packetize_stream(); codec segments = 8 image lines → line-accurate CUC datation; SEQ_FIRST/CONT/LAST grouping) under measurements/d{DD}/b{BB}/{isp, isp_offsets, packet_data_length}; SAD telemetry goes to conditions/anc_data/s{APID}/isp. Achieved per-band compression ratios replace the static compression_rate metadata. With store_decoded=False the decoded band{BB} arrays are omitted — the product then stores ISPs only, mirroring the ESA S2 Synthetic L0 (SentiWiki: L0 = compressed ISPs; ground L1A decompresses via read_l0_isp_dn()).

jobs > 1 fans the per-band CPU work (compression/packetization/SAD — _band_payload()) out to that many worker processes; the zarr writes stay in this process and the product is bit-identical to a serial run.

Return type:: str

s2_msi_raw_generator.metadata module#

Shared STAC / acquisition-identity metadata helpers.

Structure- and value-level parsing that io, naming, inventory and import_l0 all need for the same ESA-product quirks (the doubled stac_discovery nesting, "null" datetimes, unfilled orbit XPaths, and the data-take id orbit fallback). The helpers only normalise structure and coerce values; the field-preference order (datetime vs start_datetime first, etc.) is left to each caller because those orders are intentionally different.

Depends only on the standard library so it never introduces an import cycle within the package.

s2_msi_raw_generator.metadata.DATATAKE_RE = re.compile('GS2[ABC]_[A-Z0-9_]+?_(?P<orbit>\\d{6})_N')#: Recovers the 6-digit absolute orbit embedded in EOPF data-take / datastrip id (GS2A_..._<orbit>_N...).

s2_msi_raw_generator.metadata.absolute_orbit_from_datatake_id(dtid)#

Recover the absolute orbit from EOPF data-take / datastrip id, or None.

Return type:: int | None

s2_msi_raw_generator.metadata.coerce_int(value)#

Coerce to int; return (value, unparseable).

A None input is treated as simply absent — (None, False) — whereas a present but non-integer value (e.g. an unfilled @@...@@ XPath token) yields (None, True) so callers can distinguish “missing” from “malformed”.

Return type:: tuple[int | None, bool]

s2_msi_raw_generator.metadata.normalise_stac(attrs)#

Flatten root attributes to (stac, properties, flags).

Handles the doubled stac_discovery.stac_discovery nesting seen in some ESA products, recording "double_stac_discovery" in flags when it is collapsed. Always returns plain dicts, even when the metadata is missing or malformed.

Return type:: tuple[dict, dict, list[str]]

s2_msi_raw_generator.metadata.parse_iso(value)#

Parse an ISO-8601 timestamp to an aware UTC datetime; None/"null" yield None.

Return type:: datetime | None

s2_msi_raw_generator.naming module#

EOPF product file naming — the PSFD §3 product identification convention.

Implements the EOPF Product Structure and Format Definition (PSFD) §3 “Product Types and File Naming Rules” scheme:

MMMSSSCCC_YYYYMMDDTHHMMSS_UUUU_PRRR_XVVV[_Z*]

for the products this generator emits (Synthetic L0 RAW, plus the L1A / L1B / ISP / SAD context types). ECSS-M-ST-40C requires a project-wide unique product identification coding system; this module is that system for the reverse E2ES (the interface is recorded in docs/icd.md).

The grammar, field by field:

MMMSSSCCC — the 9-character type code (mission S02 + sensor + processing level); see TYPE_CODES. A code ending in the pad character _ yields the doubled underscore seen in operational product names, e.g. S02MSIL0__....
YYYYMMDDTHHMMSS — acquisition start (UTC).
UUUU — acquisition duration in whole seconds (4 digits, 0001–9999).
PRRR — platform unit letter P + relative orbit RRR (001–143).
XVVV — consolidation flag X + a 3 hex-digit discriminator VVV.
_Z* — an optional free token (segment / tile / role).

Examples (from the PSFD): S02MSIL0__20230216T182840_0001_A123_T000 and S02MSIL1C_20230629T063559_0000_A064_T3A5.

s2_msi_raw_generator.naming.CONSOLIDATION_FLAGS = frozenset({'S', 'T', '_'})#

T test / _ nominal / S special.

Type:: Valid consolidation flags (the X field)

s2_msi_raw_generator.naming.DEFAULT_START = datetime.datetime(2024, 4, 3, 10, 24, 15, tzinfo=datetime.timezone.utc)#: Fallback acquisition context used by from_l1a_context() when L1A metadata is missing (the datation reference epoch, relative orbit 45, unit A).

s2_msi_raw_generator.naming.EXTENSIONS = frozenset({'', '.zarr', '.zarr.zip'})#: Permitted product file extensions.

s2_msi_raw_generator.naming.TYPE_CODES: dict[str, str] = {'S02MSIDCA': 'MSI dark calibration (DASC)', 'S02MSIISP': 'MSI CCSDS instrument source packets', 'S02MSIL0_': 'MSI Level-0 RAW', 'S02MSIL1A': 'MSI Level-1A', 'S02MSIL1B': 'MSI Level-1B', 'S02MSIL1C': 'MSI Level-1C', 'S02MSIL2A': 'MSI Level-2A', 'S02MSISCA': 'MSI sun-diffuser calibration (ABSR)', 'S02SADISP': 'Satellite ancillary data packets'}#: The 9-character product type codes emitted by this generator (PSFD §3), mapped to a short label. The trailing _ is the PSFD pad character (used when the level mnemonic is shorter than the 3-character CCC field), so e.g. S02MSIL0_ plus the _ field separator reads S02MSIL0__.

s2_msi_raw_generator.naming.acquisition_context(attrs)#

Extract acquisition identity fields from L1A/L1B root attributes.

Reads the acquisition start, relative orbit and platform unit from the STAC discovery metadata (attrs["stac_discovery"]["properties"]). Any field that is absent falls back to a module default (DEFAULT_START, DEFAULT_RELATIVE_ORBIT, DEFAULT_UNIT) and is recorded under "derived_from_defaults".

Return type:: dict

s2_msi_raw_generator.naming.from_l1a_context(attrs, *, n_lines, line_period_s, product_type, ext='.zarr')#

Derive a product name from an L1A / L1B product’s root attributes.

Reads the acquisition context via acquisition_context() and computes the duration as n_lines * line_period_s.

Parameters:

attrs (dict) – The source product’s root attributes (STAC discovery under "stac_discovery").
n_lines (int) – Number of acquired lines (sets the duration together with line_period_s).
line_period_s (float) – Line period in seconds.
product_type (str) – The 9-character type code for the derived product (see TYPE_CODES).
ext (str) – File extension. Default ".zarr".

Returns:

The product name and an info dict {"derived_from_defaults": [field, ...]} naming each STAC field that fell back to a default.

Return type:

tuple[str, dict]

s2_msi_raw_generator.naming.parse_psfd_name(name)#

Parse a PSFD §3 product file name into its fields — the inverse of psfd_name().

Parameters:: name (str) – A product file name, with or without an extension and with or without the optional _<z_suffix> token.
Returns:: Keys product_type, start_utc (a naive datetime.datetime on UTC), duration_s (int), unit, relative_orbit (int), consolidation, discriminator, z_suffix (str or None) and ext.
Return type:: dict
Raises:: ValueError – If name does not match the grammar or carries an unknown type code.

s2_msi_raw_generator.naming.psfd_name(product_type, start_utc, duration_s, *, unit='A', relative_orbit, consolidation='T', discriminator=None, z_suffix=None, ext='.zarr')#

Build a PSFD §3 product file name.

Parameters:

product_type (str) – The 9-character type code; must be one of TYPE_CODES.
start_utc (datetime) – Acquisition start time. Timezone-aware values are converted to UTC; naive values are formatted as given.
duration_s (float) – Acquisition duration in seconds, rounded half-up to a whole second and clamped to a minimum of 1. A value rounding above 9999 is rejected.
unit (str) – Platform unit — a single upper-case letter (the P field). Default "A".
relative_orbit (int) – Relative orbit number in 1..143 (the Sentinel-2 repeat cycle).
consolidation (str) – Consolidation flag (the X field); one of CONSOLIDATION_FLAGS. Default "T".
discriminator (str | None) – Three upper-case hex characters (the VVV field). When None (default) a deterministic value is derived from the other fields via zlib.crc32.
z_suffix (str | None) – Optional free token appended as _<z_suffix> (letters, digits and underscores only).
ext (str) – File extension; one of EXTENSIONS. Default ".zarr".

Returns:

The composed product file name.

Return type:

str

Raises:

ValueError – If any field is invalid (unknown type code, out-of-range orbit or duration, or a malformed unit, consolidation, discriminator, z_suffix or extension).

Examples

>>> from datetime import datetime
>>> psfd_name("S02MSIL0_", datetime(2024, 4, 3, 10, 24, 15), 33.0,
...           relative_orbit=45, discriminator="1A2")
'S02MSIL0__20240403T102415_0033_A045_T1A2.zarr'

s2_msi_raw_generator.quality module#

L0 quality flags — msi-processor QAFlag-compatible seeds + Sentinel-2 MSK_QUALIT masks.

Two representations, one source of truth:

l0_flags() → uint16 bitmask using the same bit values as the processor’s msi_processor.computing.common.types.QAFlag (NO_DATA=1, LOST_PACKET=2, SATURATED=4, DEFECTIVE=8), so the processor’s monotone OR-accumulation picks up the generator’s seeds directly (the open-container quality/l0_flags/<band>).
to_msk_qualit() → uint8 in the Sentinel-2 MSK_QUALIT 8-bit-plane layout (the canonical Synthetic L0 product mask quality/d{DD}/b{BB}/mask).

s2_msi_raw_generator.quality.from_s10_qa(qa_s10)#

Map the low-level reverse.s10_inject_defects qa (bit0 = dead, bit1 = hot) to QAFlag bits.

Return type:: ndarray

s2_msi_raw_generator.quality.l0_flags(dn, *, dead_cols=(), hot_pixels=(), dn_max=4095)#

QAFlag-compatible (uint16) L0 quality seed for one (line, detector) frame.

Flags SATURATED where DN ≥ dn_max, NO_DATA where DN == DN_NODATA (0), DEFECTIVE on dead_cols / hot_pixels, and LOST_PACKET on wholly-zero trailing lines (the processor’s line-loss convention).

Return type:: ndarray

s2_msi_raw_generator.quality.to_msk_qualit(qa)#

Translate a QAFlag uint16 seed to the S2 MSK_QUALIT uint8 8-bit-plane mask.

Return type:: ndarray

s2_msi_raw_generator.quality_report module#

EOPF EOQC-style per-product quality report (machine-readable OK/KO JSON).

Emulates the minimum content of the EOPF eopf.qualitycontrol (EOQC) per-product report — product name/type, station, facility, sensing times, orbit numbers, an overall flag and a per-check list — self-asserted from the Synthetic L0 product the generator just wrote. Pure json/stdlib (no eopf dependency); when the EOQCProcessor is available (in the SDE) it may additionally be run. ECSS-Q-ST-20C.

s2_msi_raw_generator.quality_report.build_qc_report(root_metadata, *, product_name, software_version='0.3.0', inspection_time=None, has_measurements=True)#

Build the EOQC per-product QC report from the L0 root metadata. Overall OK iff all checks pass.

Return type:: dict

s2_msi_raw_generator.quality_report.write_qc_report(path, report)#

Write the QC report as standalone JSON (QC_report_{name}.json convention). Returns the path.

Return type:: Path

s2_msi_raw_generator.quicklook module#

Dependency-free quicklook PNG writer for the L0 / L1B products (numpy + stdlib only).

Renders a small RGB preview of a {band: 2-D array} product with a per-channel percentile contrast stretch — for the repo README / documentation front page. No matplotlib / PIL dependency (a minimal PNG encoder using zlib + struct), so it runs anywhere the generator does, including the SDE.

s2_msi_raw_generator.quicklook.save_rgb(bands, out_png, *, rgb=('B04', 'B03', 'B02'), upscale=1)#

Write an RGB quicklook PNG from a {band: 2-D array} product. Returns the path.

rgb selects the three bands for the R/G/B channels (each independently percentile-stretched). upscale nearest-neighbour enlarges small demo frames for visibility.

Return type:: str

s2_msi_raw_generator.reverse module#

Reverse radiometric chain (L1B radiance → Synthetic L0 RAW DN), pure NumPy.

Each function is one ATBD §5 step (S1, S6, S7, S11, S12, S13, S14). All operate on a 2-D per-band, per-detector array (lines, detector_columns). The MVP chain (Increment 1) is S1 → S6 → S7 → S11 → S12 → S13 → S14. Steps S3/S4/S5/S8/S9/S10/S15 (framing, offset, binning, SWIR rearrangement, crosstalk, blind pixels, ISP) are Increment 3+/Full.

s2_msi_raw_generator.reverse.forward_radiometric(dn, adf)#

Algebraic forward (processor) inverse of reverse_radiometric(): DN_raw → radiance.

Return type:: ndarray

s2_msi_raw_generator.reverse.reverse_full(radiance, adf, rng, *, swir_shifts=None, dead_cols=(), hot_pixels=())#

Extended per-band reverse chain (Inc 3): adds S8 (SWIR re-arrangement, reverse) and S10 (defects).

S1 → S6 → S7 → [S8] → S13 → S11 → S12 → [S10] → S14. Noise (S13) is on the signal DN (before the dark pedestal); defects (S10) are applied last so dead columns stay 0 and hot pixels stay saturated in the output. (S5 un-bin changes width and S9 crosstalk needs multiple bands — applied separately.) Returns (uint16 Synthetic L0 DN, qa uint8).

Return type:: tuple[ndarray, ndarray]

s2_msi_raw_generator.reverse.reverse_mvp(radiance, adf, rng)#

Full MVP reverse chain for one band/detector: radiance → uint16 Synthetic L0 DN.

S1 → S6 → S7 → S13 → S11 → S12 → S14. Noise (S13) is impressed on the signal DN (before the dark pedestal is added in S11), so σ=√(α²+β·DN_signal) reproduces the SNR@Lref exactly — the α,β model already captures the total noise at a given signal level.

Return type:: ndarray

s2_msi_raw_generator.reverse.reverse_radiometric(radiance, adf)#

Reverse S1→S7→S11→S12 only (no PSF, no noise, no quantize) — exactly invertible.

Return type:: ndarray

s2_msi_raw_generator.reverse.s10_inject_defects(img, dead_cols=(), hot_pixels=(), dn_max=4095)#

S10 — re-insert blind/defective pixels: dead columns → 0, hot pixels → saturated.

Returns (img_with_defects, qa) where qa (uint8) has bit0 = dead, bit1 = hot. (S2C Cal/Val: 3 defective in B11, 1 in B12.)

Return type:: tuple[ndarray, ndarray]

s2_msi_raw_generator.reverse.s11_reapply_dark(dn, dark_dn)#

S11 — add the per-detector dark signal back: DN += dark.

Return type:: ndarray

s2_msi_raw_generator.reverse.s12_reapply_onboard_eq(dn, eq_gain, eq_offset)#

S12 — reverse onboard equalization: DN_raw = DN_eq / gain_ob + offset_ob.

Return type:: ndarray

s2_msi_raw_generator.reverse.s13_add_noise(dn, noise_a, noise_b, rng)#

S13 — add signal-dependent sensor noise: σ = √(a + b·DN), DN += N(0, σ).

Return type:: ndarray

s2_msi_raw_generator.reverse.s14_quantize(dn, dn_max=4095)#

S14 — quantize to 12-bit and clip: clip(round(DN), 0, 4095) → uint16.

Return type:: ndarray

s2_msi_raw_generator.reverse.s1_radiance_to_dn(radiance, gain)#

S1 — radiance → equalized signal DN: DN = A · L, the absolute-calibration term of the official L1 ATBD raw model X = A·G·L + D. A is Band.cal_gain (datasheet-derived from the noise model + SNR@Lref, so the chain reproduces SNR@Lref). Processor forward: L = DN/A.

Return type:: ndarray

s2_msi_raw_generator.reverse.s3_undo_framing(img)#

S3 — undo scene framing & round/clamp.

Framing (cutting the continuous detector strip into scenes) is a product-assembly concern, handled in s2_msi_raw_generator.l0product; on a single full-strip granule array this is identity.

Return type:: ndarray

s2_msi_raw_generator.reverse.s4_undo_radiometric_offset(dn, offset=0.0)#

S4 — remove the PB04.00 product offset: DN -= offset.

Only relevant when entering from offset-encoded product DN; for the radiance entry path the offset is 0 (raw counts carry no product offset). sensor.RADIO_ADD_OFFSET_L1B is −100.

Return type:: ndarray

s2_msi_raw_generator.reverse.s5_unbin(img, factor=3, axis=1)#

S5 — un-bin a 60 m band to detector-level by replication along axis (×``factor``).

Inverse of forward mean-binning (e.g. 20 m → 60 m, factor 3). Replication preserves each cell’s value; binning the result back recovers the input.

Return type:: ndarray

s2_msi_raw_generator.reverse.s6_psf_reblur(img, psf)#

S6 — re-introduce optical blur: circular 2-D convolution with a DC-unit PSF kernel.

Radiometry-preserving (kernel sums to 1 ⇒ array total is preserved exactly).

Return type:: ndarray

s2_msi_raw_generator.reverse.s7_impress_relative_response(dn, prnu_gain)#

S7 — re-introduce per-detector PRNU: DN /= gain[detector] (undo equalization).

Return type:: ndarray

s2_msi_raw_generator.reverse.s8_restage_swir(img, shifts)#

S8 — SWIR re-arrangement (reverse): re-introduce the staggered SWIR readout by rolling each detector column along-track.

shifts[c] is the integer line shift for column c (PyRawS-style deterministic shift map). Exactly invertible by restaging with -shifts.

Return type:: ndarray

s2_msi_raw_generator.reverse.s9_apply_crosstalk(bands, coeff=0.002)#

S9 — inter-band optical/electrical crosstalk within one resolution group.

Each band gains coeff × (sum of the other bands). bands must be a dict of identical- shape arrays. <0.5 % channel-to-channel (ATBD Annex A.6). For small coeff the inverse is ≈ subtract coeff × neighbours.

Return type:: dict[str, ndarray]

s2_msi_raw_generator.s3fetch module#

Anonymous S3 object fetcher (stdlib only) for retrieving public input products.

A dependency-free helper for pulling Sentinel-2 L1A / L1B granules (or any objects) from a public, anonymously-readable S3-compatible endpoint — e.g. an open-data bucket — into a local directory, with listing pagination and per-object integrity verification. It uses only the standard library (urllib.request, xml.etree.ElementTree, hashlib, concurrent.futures), in keeping with the project’s pure numpy + stdlib policy, and issues unauthenticated GET requests only (no credentials, no request signing).

Integrity is checked on every download: the HTTP Content-Length must equal the listed object size, and for single-part objects (an ETag without a - part-count suffix) the MD5 of the downloaded bytes must equal the ETag. Downloads are written to a .part-tmp sidecar and atomically os.replace()-d into place, and destination paths are confined to the target directory (keys that would escape via a parent reference are rejected).

class s2_msi_raw_generator.s3fetch.S3Object(key, size, etag)#

Bases: object

One object in an S3 listing (a single Contents entry).

etag: str#: ETag with surrounding quotes removed; a - marks a multipart upload (not a plain MD5).

key: str#: The object key (its path within the bucket).

size: int#: Object size in bytes.

s2_msi_raw_generator.s3fetch.fetch_prefix(endpoint, prefix, dest_dir, *, strip_prefix=None, jobs=4, resume=True, timeout=120.0)#

Download every object under prefix into dest_dir with integrity verification.

Each object is written to dest_dir / <key with strip_prefix removed>. Downloads go to a .part-tmp sidecar first and are atomically os.replace()-d into place. The HTTP Content-Length is checked against the listed size for every object, and single-part ETags are verified as MD5. With resume set, an existing file whose size (and, when checkable, MD5) already matches is left untouched.

Parameters:

endpoint (str) – Bucket endpoint URL.
prefix (str) – Key prefix to fetch.
dest_dir (str | Path) – Local destination directory (created as needed). Keys that would resolve outside it are rejected.
strip_prefix (str | None) – The portion removed from the front of each key to form its local path. Defaults to prefix.
jobs (int) – Number of concurrent download threads. Default 4.
resume (bool) – Skip files that are already complete. Default True.
timeout (float) – Per-request timeout in seconds. Default 120.

Returns:

A manifest with keys endpoint, prefix, n_objects, total_bytes, files and errors. Each files entry is {"key", "size", "etag", "verify"} where verify is one of "md5", "size" or "skip".

Return type:

dict

Raises:

RuntimeError – If any object failed to download or verify (or was rejected as an unsafe path). Every other object is processed first; the manifest is attached to the exception as exc.manifest.

s2_msi_raw_generator.s3fetch.list_prefix(endpoint, prefix, *, timeout=60.0)#

List every object under prefix at an S3-compatible endpoint (ListObjectsV2).

Follows NextContinuationToken pagination until the listing is exhausted.

Parameters:

endpoint (str) – Bucket endpoint URL, e.g. "https://host/bucket".
prefix (str) – Key prefix to list.
timeout (float) – Per-request timeout in seconds. Default 60.

Returns:

Every listed object under prefix.

Return type:

list[S3Object]

s2_msi_raw_generator.s3fetch.parse_list_xml(xml_bytes)#

Parse an S3 ListBucketResult (ListObjectsV2) response.

The parse is namespace-tolerant — the default S3 XML namespace on every tag is stripped.

Parameters:: xml_bytes (bytes) – The raw XML response body.
Returns:: The objects on this page and the NextContinuationToken for the following page, or None when the listing is not truncated.
Return type:: tuple[list[S3Object], str | None]

s2_msi_raw_generator.s3fetch.save_manifest(manifest, path)#

Write a fetch manifest to path as indented JSON.

Return type:: None

s2_msi_raw_generator.sad module#

Satellite Ancillary Data (SAD) — AOCS attitude + orbit ephemeris + thermal for the L0 ISP.

The S2A S2A_OPER_AUX_SADATA_* / HKTM binary inner layout is proprietary (S2 PDGS ISP ICD, not public), so a bit-exact decode is infeasible. This module degrades gracefully in three tiers:

Outer framing decode (optional): scan_ccsds_packets() / decode_sadata_framing() parse the CCSDS Space-Packet boundaries (APID, length, CUC time) of a SADATA/HKTM stream.
Inner AOCS/orbit synthesis (physically plausible values): synth_orbit_attitude() propagates a Sentinel-2 sun-synchronous circular orbit (ECEF position/velocity), a nadir/velocity-aligned attitude quaternion, and a slow thermal cycle — numpy-only, no external ephemeris.
Re-pack as CCSDS ISP (pack_sad_isp()): a documented big-endian float64 payload [q0..q3, x, y, z, vx, vy, vz, T] replacing the previous all-zero SAD payload.

msi-processor does not consume the SAD for L1B (it is optional passthrough); this exists for product fidelity, EOQC Datation_Sync/Time_Correlation and future rigorous geometry.

class s2_msi_raw_generator.sad.AocsSeries(times_s, position, velocity, quaternion, thermal)#

Bases: object

Per-sample AOCS/orbit series (ECEF), one row per SAD packet or image line.

position: ndarray#

quaternion: ndarray#

thermal: ndarray#

times_s: ndarray#

velocity: ndarray#

s2_msi_raw_generator.sad.aocs_to_conditions(aocs)#

Per-line conditions/* arrays for the open-container L0 handoff (used by the E2E MR).

Return type:: dict

s2_msi_raw_generator.sad.decode_sadata_framing(tar_path)#

Tier-1 decode: CCSDS packet framing of an S2A_OPER_AUX_SADATA_* / HKTM .tar.

Returns the aggregated packet headers (APID, sequence, length, CUC not decoded here — inner layout is proprietary). Enrichment only; the synthesis path (synth_orbit_attitude()) is the default.

Return type:: list[dict]

s2_msi_raw_generator.sad.orbit_ephemeris(datation, n_lines)#

(start, stop) orbit-ephemeris blocks (TAI/UTC/UT1 + ECEF position/velocity) for the metadata.

Return type:: tuple[dict, dict]

s2_msi_raw_generator.sad.pack_sad_isp(aocs, apid)#

Pack an AocsSeries into CCSDS SAD ISP packets (primary header + CUC + payload).

Return type:: tuple[ndarray, ndarray]

s2_msi_raw_generator.sad.scan_ccsds_packets(buf)#

Scan a byte buffer for consecutive CCSDS Space Packets (outer-framing decode, tier 1).

Return type:: list[dict]

s2_msi_raw_generator.sad.synth_orbit_attitude(times_s, *, a=7157000.0, inclination_deg=98.62, raan_deg=0.0, u0_deg=0.0)#

Physically-plausible S2 circular-orbit ECEF position/velocity + nadir attitude at times_s.

Return type:: AocsSeries

s2_msi_raw_generator.sad.unpack_sad_payload(rec)#

Decode one SAD ISP record → {time, quaternion, position, velocity, thermal}.

Return type:: dict

s2_msi_raw_generator.sensor module#

Sentinel-2 MSI sensor model — REAL values harvested from product metadata + official docs.

Sources: ATBD Annex A.11 (physical_gains, TDI, line_period from S02MSIL1B_20240403…), SentiWiki MSI radiometric table (SNR@Lref) and the official Spectral Response Functions document (COPE-GSEG-EOPG-TN-15-0007 v4.0, 2024) for per-unit band centre/bandwidth/equivalent wavelength. No synthetic values here. Per-unit (S2A/S2B/S2C) data is matched to the product’s satellite via unit_from_platform().

class s2_msi_raw_generator.sensor.Band(name, gsd_m, physical_gain, lref, snr_at_lref, has_tdi, unit='S2A', centre_nm=0.0, bandwidth_nm=0.0, equiv_wavelength_nm=0.0, noise_alpha=0.0, noise_beta=0.0)#

Bases: object

Per-band sensor parameters (spectral characterisation is for unit unit).

bandwidth_nm: float = 0.0#

property cal_gain: float#

dn_ref / Lref — datasheet-derived (noise α,β + SNR@Lref), so the chain reproduces the SNR@Lref.

Type:: Absolute calibration gain A used in S1 (DN = A·L)

centre_nm: float = 0.0#

property dark_dsnu: float#: Per-pixel dark non-uniformity (1σ DN) for this band’s focal plane (DQR value).

property dn_ref: float#

Equalized signal DN at Lref, on the true 12-bit instrument scale.

Derived from the REAL noise model + REAL SNR@Lref: the DN where the noise σ=√(α²+β·DN) gives the spec SNR (DN/σ = SNR), i.e. the positive root of DN² − SNR²β·DN − SNR²α² = 0. This anchors the chain so the α,β reproduce the SNR@Lref. (The product’s physical_gain is incoherent with α,β on this synthetic dataset, so it is kept for metadata / the round-trip bridge but not used to set the working DN scale.)

equiv_wavelength_nm: float = 0.0#

gsd_m: int#

has_tdi: bool#

lref: float#

name: str#

noise_alpha: float = 0.0#

noise_beta: float = 0.0#

physical_gain: float#

snr_at_lref: float#

unit: str = 'S2A'#

s2_msi_raw_generator.sensor.all_bands(unit='S2A')#

All 13 bands in canonical order, for unit S2A/B/C.

Return type:: list[Band]

s2_msi_raw_generator.sensor.band(name, unit='S2A')#

Return the Band model for a band name (e.g. "B04"), for unit S2A/B/C.

Return type:: Band

s2_msi_raw_generator.sensor.band_number(name)#

Band number used in the Synthetic L0 spectral_band_info keys: B01 → "01", B8A → "8A".

Return type:: str

s2_msi_raw_generator.sensor.esun(name, unit='S2A')#

Per-band ESUN (extraterrestrial solar irradiance, W·m⁻²·µm⁻¹) for unit — Thuillier 2003.

The value the processor’s toa unit consumes as the spectral ADF for TOA reflectance. Only S2A/S2B are published (ATBD §A.3); Sentinel-2C raises until its SRF-specific set exists.

Return type:: float

s2_msi_raw_generator.sensor.spectral_band_info(unit='S2A')#

Per-band spectral_band_info block for the L0 root metadata (values).

Radiometric values are from product metadata (Annex A.11); spectral centre/bandwidth/equivalent wavelength are the per-unit values from the SRF document (COPE-GSEG-EOPG-TN-15-0007).

Return type:: dict[str, dict]

s2_msi_raw_generator.sensor.unit_from_platform(platform)#

Map a product platform string ("Sentinel-2A", "S2B", …) to a unit key S2A/B/C.

Return type:: str

s2_msi_raw_generator.sensor.zarr_band_key(name)#

Zarr group key for a band in the Synthetic L0 product: B03 → "b03", B8A → "b8a".

Return type:: str