Interface control document

Introduction

This section constitutes the Interface control document (ICD) for the eo-data-embedding project.

It defines the public interfaces of the software.

Software overview

eo-data-embedding is a ground-segment, non-flight Python library that turns Sentinel-1/2 imagery into frozen foundation-model embeddings and exposes them to four downstream tasks — similarity search, few-shot linear probing, and bitemporal change detection — over a shared Parquet embedding store. The full software overview is given in the SRS; this ICD describes only the interfaces. The DRD reference for this project is ECSS-E-ST-40C Rev.1, Annex E; because the EOPF knowledge pool has no dedicated ICD template, the CPM API and the PSFD (Product Structure and Format Definition) serve as the normative interface reference.

Interface design (implemented)

All interfaces are Python functions/classes in eo_data_embedding/. The canonical contract shared across every phase is an embedder exposing encode(x) -> (B, D) and a Parquet embedding store that decouples the heavy GPU embedding pass from the cheap downstream tasks (search/probe/change).

Internal interfaces (module contracts)

Module	Key entry points	Contract
embed	ViTEmbedder, ClayEmbedder, load_embedder(name, **kw)	encode(x) -> (B, D) float32 CPU. Clay: raw (B,C,H,W) → (B,1024) class token (or (B,P,1024) patches with return_patches=True); ViT-small sanity → (B,384). Frozen encoders.
store	save_embeddings(path, ids, vectors, modality, labels=None), load_embeddings(path), stack_vectors(df)	Parquet, one row/tile, schema {id, modality, vector:list<float32>, [label]}; stack_vectors → dense (N,D).
search	build_index(vectors, normalize=True), search(index, queries, top_k=12), retrieval_metrics(neigh_labels, query_labels, class_total=None)	faiss.IndexFlatIP (cosine via L2-norm + inner product); metrics → {precision, recall, map, k}.
probe	heldout_split, sample_shots, linear_probe_multi, full_probe, train_probe, LinearProbe, save_probe/load_probe	Few-shot linear classifier on frozen (N,D) embeddings; probe persisted as version-independent .npz (coef, intercept, classes).
change	embedding_change_score, pick_threshold, binary_change_metrics, patch_change_map, delta_features, tile_image, tile_mask_labels, patch_mask_labels	Bitemporal change: per-tile/per-patch scores from (N,D)/(P,D); threshold picked on train split; metrics {f1, precision, recall, iou, kappa, accuracy, roc_auc, threshold}.
config	load_config(path), cfg_get(cfg, dotted, default)	YAML (configs/default.yaml) → argparse defaults; CLI flags override.

External interfaces

• Data (data.py, via TorchGeo + webdataset): EuroSAT (S2 13→10 Clay bands), BigEarthNet-MM (S1+S2), SSL4EO-S12 v1.1 (streaming, paired S1/S2), OSCD (bitemporal S2). Raw un-normalized pixels; bands reordered to Clay order (clay_metadata.py). S2 Clay order = 10 bands @10 m; S1 = vv,vh (dB) @10 m.

• Model (Clay v1.5 — fetched, not vendored): clay-v1.5.ckpt from HF made-with-clay/Clay; claymodel from git; metadata.yaml (pinned commit) resolved via metadata_path/CLAY_METADATA/configs/clay/. Fixed image_size=256, dim 1024, frozen, fp32; datacube keys pixels,time(B,4),latlon(B,4),gsd,waves (time/latlon zeros).

• Storage: embeddings → Apache Parquet (artifacts/embeddings.parquet); FAISS index in-memory (rebuilt per run); probe → .npz; reports → Markdown.

Known gaps (carried from source): CRS/projection not asserted (pixel-array only); FAISS index not serialized to disk; Prithvi optical fallback declared but NotImplementedError.

Requirements and design

General provisions to the requirements in the IRD

Not applicable as a separately identified requirement set. This non-flight ground ML library has no IRD; its interfaces are not derived from numbered higher-level interface requirements but from the implemented Python module contracts (above) and from the external standards they consume (the EOPF CPM API/PSFD, the Clay v1.5 model contract, and the Sentinel/EuroSAT/OSCD dataset formats). Configuration control of those interfaces is handled by the SDP and the pinned dependency set in pyproject.toml.

Interface requirements

The software item interfaces are fully described above and summarised here against the ECSS interface categories:

1. Software-to-software interfaces. Internally, the modules are wired as a pipeline: embed produces float32 CPU embeddings that store persists to Parquet, and store feeds search, probe, and change. The canonical contract is an embedder exposing encode(x) -> (B, D) and the Parquet store schema {id, modality, vector:list<float32>, [label]}, which decouples the GPU embedding pass from the downstream CPU tasks (see the internal-interfaces table above). Externally, the software item reuses the EOPF CPM (EOProduct, Zarr/SAFE/NetCDF I/O), the frozen Clay v1.5 encoder, TorchGeo dataset loaders, and FAISS — described in External interfaces above.

2. Software-to-hardware interfaces. None beyond an optional CUDA-capable GPU used only to accelerate the embedding pass; the library is otherwise CPU-only and has no direct hardware, signal, or telemetry/telecommand interfaces (it is a non-flight ground tool).

3. Man–machine interfaces. A console entry point (eo-data-embedding / eoemb) drives the phase scripts, and an optional Gradio UI provides interactive search. Both are thin wrappers over the module contracts above; the SUM documents their usage.

4. Database structure. The only persistent data structure is the Apache Parquet embedding store (schema above); there is no relational database, signal definition, or TM/TC plan.

Error behaviour. Interface-level error handling is by Python exceptions: unimplemented backbones (prithvi) raise NotImplementedError; a missing config file makes load_config return {} so callers fall back to argparse defaults; and load_embeddings /build_index propagate I/O and dimension errors to the caller rather than masking them.

Timing. No hard real-time or timing requirements apply; this is a batch ground-processing library.

Interface design

The external interface design is given above (External interfaces); the design of each interface — provided service, data item names/types/dimensions, and ranges — is detailed per data item below. There is no TM/TC plan, command/telemetry stream, or wire protocol: all interfaces are in-process Python calls plus on-disk artifacts, so “physical interface architecture” reduces to the function signatures and file formats already specified.

Because this is a high-reuse integration project (see the SRF), the load-bearing external interfaces are reused, third-party contracts referenced by their upstream documentation:

External interface	Provided service	Data item — type, dimension, range / initial value	Source / destination
Sentinel-2 optical input (EuroSAT, OSCD, BigEarthNet, SSL4EO-S12)	Raw multispectral reflectance tiles	s2 float tensor (N, 10, H, W), bands in Clay order blue,green,red,rededge1,rededge2,rededge3,nir,nir08,swir16,swir22; central wavelengths [0.493,0.56,0.665,0.704,0.74,0.783,0.842,0.865,1.61,2.19] µm; GSD 10 m; un-normalized	TorchGeo / webdataset loaders (data.py) → embed
Sentinel-1 SAR input (BigEarthNet, SSL4EO-S12)	Raw backscatter tiles	s1 float tensor (N, 2, H, W), bands vv,vh (RTC backscatter, dB); GSD 10 m	TorchGeo / webdataset loaders → embed
OSCD bitemporal pair	Change-detection scene pair + mask	{id, t1 (10,H,W), t2 (10,H,W), mask (H,W) ∈ {0,1}}	oscd_pairs(...) → change
Clay v1.5 model weights	Frozen ViT-MAE encoder	clay-v1.5.ckpt (HF made-with-clay/Clay); fixed image_size=256, embedding dim 1024, fp32; datacube keys pixels, time (B,4), latlon (B,4), gsd, waves (time/latlon initialised to zeros)	HuggingFace + claymodel (git f14e698) → ClayEmbedder
Clay metadata	Band/wavelength/mean/std/GSD table	metadata.yaml, pinned Clay commit f14e698; resolved via metadata_path / CLAY_METADATA / configs/clay/metadata.yaml; mirrored in clay_metadata.py	Clay repo → ClayEmbedder
Embedding store	Persisted embeddings	Apache Parquet, schema {id:str, modality:str, vector:list<float32> (length D), label?:int}, one row per tile; default path artifacts/embeddings.parquet	store.save_embeddings ↔ store.load_embeddings
FAISS index	Exact inner-product retrieval index	faiss.IndexFlatIP of dim D = vectors.shape[1]; vectors L2-normalized so inner product = cosine; in-memory, rebuilt per run (no on-disk format)	search.build_index ← Parquet store
Saved linear probe	Version-independent classifier weights	NumPy .npz with coef (n_classes, D), intercept (n_classes,), classes; no pickled estimator	probe.save_probe ↔ probe.load_probe
Reports	Human-readable results	Markdown — artifacts/search_results.md, artifacts/probe_results.md	phase scripts

The communication protocol for the reused EOPF, Clay, TorchGeo and FAISS interfaces is by reference to their upstream documentation (CPM API/PSFD, Clay model card, TorchGeo dataset API, FAISS API). Specific design requirements for intended reuse are captured in the SRF.

Validation requirements

This ICD does not introduce its own numbered interface-requirement set, so no per-requirement validation matrix is maintained here. The interface contracts above are validated empirically by the project’s unit and integration test suite: the encode(x) -> (B, D) shape/dtype contracts, the Parquet store round-trip (save_embeddings/load_embeddings/stack_vectors), the FAISS index build/search, the .npz probe persistence, and the dataset band-reordering mappings are each covered by tests run in CI. Reused external interfaces (EOPF CPM, Clay, TorchGeo, FAISS) are validated against their upstream releases and the pinned dependency set. The overall validation approach and its requirement correlation are owned by the SVS/SVR and the SDP, not duplicated in this ICD.

Traceability

No separate forward/backward interface-requirement traceability matrix is held in this ICD, because the document defines no numbered interface requirements of its own (see General provisions above). Each interface here traces directly to its implementing module in src/eo_data_embedding/ (embed.py, store.py, search.py, probe.py, change.py, config.py, data.py) and to the external contracts it reuses (EOPF CPM API/PSFD, the Clay v1.5 model card, the TorchGeo dataset API, and the FAISS API). Project-level requirement-to-design traceability is maintained in the DJF and the SDP; reference is made to that documentation rather than reproducing the matrices here.