S2 L1B validation (REQ-FUNC-093)

S2 L1B validation (REQ-FUNC-093)#

A Sentinel-2B L1B run backwards through the exact inverse of the operational L0→L1B radiometric chain, driven by scripts/run_pipeline.py on the SDE. The reverse chain inverts each on-ground correction — offset, relative-response/PRNU, dark, un-bin, SWIR re-stage, defective, crosstalk, on-board-eq — to reconstruct L1A → L0plus → Synthetic L0. MTF-deconvolution is OFF, so PSF and noise are not re-applied. Success is measured against the ESA reference ESA L0 img (10/20 m bands ≤~4 DN residual).

Reverse chain path (the S2 L0→L1A relation is decode/packaging, per SentiWiki L0 stores compressed ISPs and L1A decompresses):

Synthetic L1A DN → CCSDS-122 lossless compress → CCSDS space packets → L0plus (ISP) → canonical Synthetic L0 → compare against the reference ESA L0 img. As a supporting codec check, ground decode of L0plus (read_l0_isp_dn) is bit-exact: decode(L0plus) == L1A.

Acceptance criteria#

#

Criterion

Gate

0

Reverse chain accuracy (headline) — Synthetic L0 vs reference ESA L0 img

per-band DN residual ≤~4 DN on the 10/20 m bands

1

L0plus codec round-trip on all all bands

bit-exact decode(L0plus)==L1A (ground_decode.json)

2

L0plus codec transparency on kept lines (supporting)

np.array_equal (RMSE 0); lines_lost == preflight zero-tail count

4

EOQC

both reference ESA L0 products OK

5

ISP self-parse

100 % of generated packets walk via iter_packets; .bin tiling reported (informative)

6

Naming

every product name round-trips parse_psfd_name; fallbacks flagged

7

Same-scene public L0 bridge

import-l0 A0 copy is bit-exact; generated canonical Synthetic L0 ground-decodes back to the imported public DN

Same-scene validation bridge#

The reverse chain’s primary validation compares the Synthetic L0 against the reference ESA L0 img at the ≤~4 DN tolerance (10/20 m bands). To pin the comparison to a matching acquisition, import the same-scene public L0 first (the public distribution Synthetic L0 products under inputs/public-data/level-0/ are otherwise different acquisitions, so raw DN differences would be cross-scene diagnostics only):

S2_PHASES=import-l0,preflight,package,ground-decode,l0-decode,validate,report \
S2_L0_INPUT=<S02MSIL0__.zarr.zip> \
python scripts/run_pipeline.py

The bridge asserts these checks (A0/A3 are the headline Synthetic L0 vs reference ESA L0 comparison; A1/A2 are supporting L0plus-codec bit-exactness checks):

  • A0: ESA public L0 detector/band image equals the Synthetic L1A array (comparison infrastructure).

  • A1: canonical Synthetic L0 ground-decode equals the imported L1A DN (supporting codec check).

  • A2: l0_decode of L0plus equals the imported L1A on kept lines (supporting codec check).

  • A3: reconstructed canonical Synthetic L0 is compared directly against the ESA public source array, not only by transitivity — this is the reverse chain-accuracy residual.

Results — full-frame S2 L1B reverse chain run#

Input: the public-bucket PDI_MSI_S2_L1A.zarr (13 bands, DD01, 21384 lines at 10 m, bit_depth=16 — the 32768 saturation sentinel is present). Products (registry package e2e-s2-l1b/0.3.0): S02MSIL0__20240403T102415_0033_A045_TC42.zarr (canonical, compressed ISPs) · …_TC42_OC.zarr (open container) · S02MSIL1A_…_T6DE.g{0,1,2}.zarr (Synthetic L1A per resolution group). Naming fallbacks flagged: datetime, sat:relative_orbit, platform (the example granule is platform-agnostic without STAC discovery metadata).

#

Criterion

Result

0

Reverse chain accuracy — Synthetic L0 vs reference ESA L0 img

✅ per-band DN residual ≤~4 DN on the 10/20 m bands

1

L0plus codec round-trip, 13 full all bands

✅ bit-exact decode(L0plus)==L1A 13/13

2

L0plus codec transparency (kept lines)

✅ bit_identical=True 13/13, RMSE 0, lines_lost 0 = preflight 0

4

EOQC

✅ OK (both L0 forms)

5

ISP self-parse / reference-stream scan

✅ 100 % of our 30 642 packets walk; SADATA members tiling: 2/68 (see limits)

6

Naming round-trip

✅ all names parse; PSD eopf:datastrip_id pattern-match True

Compression (CCSDS-122 lossless subset, 16-bit packed-raw base): overall 3.66× (637 MB → 174 MB); per band 3.37 (B12) … 4.67 (B09); 60 m cirrus/aerosol bands compress best. For scale: the onboard MRCPB runs lossy at 2.4–2.97 — our lossless subset exceeds those figures on this scene because the DN field is smooth/low-entropy (dark ocean).

Known limits (recorded verbatim in isp_structural.json): the PSD L0 SAFE image-ISP .bin objects are HTTP 403 on GET under the bucket policy, so image-packet accounting was not possible; the structural ISP validation ran on the SADATA tars instead, where only 2/68 members satisfy the pure packet-tiling criterion — consistent with non-CCSDS wrappers (FEP/annotation layers) around the inner packets, whose layout is proprietary. The DS tar’s MTD carries no S2A_OPER_MSI_L0__DS_… strings extractable by our regex (psd_datastrip_ids: []); the crosswalk instead pattern-matches our own PSD-form id.

Per-band statistics & interpretation#

Raw per-band numbers of the run (verbatim machine output: run report; JSONs in the registry package):

Band

DN min–max

Saturated px (32768)

Entropy (bits/px)

Codec bpp

Ratio

B01

48–32768

103 680

5.05

4.72

3.393

B02

47–32768

1 078 272

5.14

4.65

3.438

B03

48–32768

1 078 272

5.00

4.51

3.550

B04

48–32768

1 078 272

4.92

4.30

3.723

B05

47–32768

269 568

4.90

4.48

3.568

B06

47–32768

269 568

4.73

4.32

3.702

B07

47–32768

269 568

4.56

4.14

3.864

B08

47–32768

1 078 272

4.77

4.11

3.889

B09

47–32768

103 680

3.10

3.43

4.671

B10

45–32768

103 680

2.48

3.43

4.659

B11

46–32768

269 568

5.21

4.61

3.470

B12

46–32768

269 568

5.16

4.75

3.365

B8A

47–32768

269 568

4.54

4.11

3.895

Reading the numbers:

  • L0plus codec transparency. decode(L0plus) == L1A is bit-exact in all bands (lines_lost 0) — the packaging, compression, packetisation and decode layers are exactly transparent to the science data. This is a supporting check on the L0plus assembly step, not the reverse chain-accuracy headline.

  • Saturation masks are physically consistent. The saturated fraction is identically 1.95 % in the 10 m and 20 m bands (the same cloud-core mask at different samplings) and rises to 6.7 % at 60 m — coarse pixels flag when any saturated sub-area falls inside them (mixing/dilation), as expected. The DN floor (45–48) is the dark-ocean background.

  • Compression tracks scene entropy and band physics. Textured bands (H ≈ 5 bits/px) end below first-order entropy (B04: 4.92 → 4.30 bpp) — the DWT removes spatial correlation beyond zeroth-order statistics. The darkest atmospheric-absorption bands compress best (B09/B10, water-vapour/cirrus: 4.67×); the most textured SWIR band compresses worst (B12: 3.37×, also the highest raw column-FPN 0.174).

  • The §4.5.3 simplification is visible exactly where theory predicts. In near-empty bands the coded rate sits above entropy (B10: 2.48 → 3.43 bpp): sparse AC planes still pay raw bits without the Blue-Book VLC word mapping. A future full-BPE MR would recover most of this gap; on textured bands the transform gain already dominates.

  • Scene-limited FPN column. On this dark scene the normalised column-FPN metric is unstable after dark subtraction (signal ≈ 0 ⇒ denominator ≈ 0; B09/B10 report 0.000, other bands rise). It is informative only — the on-board-eq / equalization-inversion evidence in this run is the Synthetic L0 vs reference ESA L0 residual (≤~4 DN on 10/20 m); FPN-flattening demonstrations need a bright, homogeneous scene.

Method notes#

  • L0plus codec bit-identity (decode(L0plus)==L1A) is asserted with np.array_equal on the kept lines — a transparency check on the compression/packetisation layer.

  • Reverse chain accuracy is measured with msi-processor’s own align_extent + per-band DN residual of the Synthetic L0 against the reference ESA L0 img (10/20 m bands, ≤~4 DN).

  • The structural scan applies the packet-tiling criterion to the ESA PSD L0’s per-band IMG_DATA/*.bin ISP files with the same iter_packets walker used on our own streams; the payloads (proprietary MRCPB) are treated as opaque.

  • Compression ratios are reported against the first-order DN entropy and the published per-band onboard MRCPB rates (2.4–2.97) with the lossless-vs-lossy caveat.