Software unit and integration test report

This page constitutes the Software unit and integration test report (SUITR) of the eo-data-embedding project.

Reports generated the 26/06/2026 00:11.

Test approach and levels

Verification follows the project Verification & Validation Plan (compliance/drd/vv-plan.md), which merges ECSS-E-ST-40C Annexes I/J/K and applies the ML V&V discipline of ECSS-E-HB-40-02A. The dominant method is test (T), executed automatically on every push and pull request, backed by inspection (I) for the static quality gates.

Two test levels are exercised:

• Unit — nine tests/test_*.py modules exercise each eo_data_embedding/<module>.py in isolation against deterministic, synthetic inputs. A shared conftest.py fixture seeds every randomised case (np.random.default_rng(0)) so runs are reproducible. Unit tests use no network and no datasets: the embedder is built with pretrained=False (random weights, no download) and the change/probe/search tests operate on in-memory NumPy/Torch tensors.

• Integration — tests/test_phase0_smoke.py drives the full synthetic pipeline end to end (encode → parquet store → FAISS → few-shot probe) by subprocess-running scripts/phase0_smoke.py. It is marked @pytest.mark.slow and deselected by default; it runs explicitly with pytest -m slow. The CPU Docker image build is the second integration check: it proves the deployable artefact assembles with a CPU-consistent torch/torchvision ABI.

Test suite and CI

The suite comprises nine test modules plus a shared conftest.py:

Test module	Module(s) under test	REQ
test_embed.py	embed.py (ViTEmbedder)	REQ-F-01
test_clay_metadata.py	clay_metadata.py (band maps, constants)	REQ-F-01
test_store.py	store.py (parquet save/load, stack_vectors)	REQ-F-02
test_search.py	search.py (FAISS build/search, retrieval_metrics)	REQ-F-03, REQ-N-02
test_probe.py	probe.py (splits, linear probe, roundtrip)	REQ-F-04
test_baseline.py	baseline.py (ResNet-18 multispectral, CNN probe)	REQ-F-04
test_change.py	change.py (Δ-score, tiling, threshold, metrics)	REQ-F-05, REQ-N-02
test_config.py	config.py (load_config, cfg_get)	REQ-N-01
test_phase0_smoke.py	full pipeline via scripts/phase0_smoke.py (slow)	REQ-F-06 (integration)

CI runs two sequential jobs on push/PR to main: lint-test (Python 3.11) installs CPU-matched torch/torchvision, then runs ruff check, ruff format --check and pytest (slow deselected); docker-cpu (needs: lint-test) builds the deployable CPU image.

Pass criteria

A change is verified when, on the pull request: (1) ruff check and ruff format --check report no findings; (2) pytest (default, slow-deselected) exits 0; and (3) the CPU Docker image builds. The slow integration smoke asserts a clean exit (returncode == 0) for the Phase-0 script. The ML-validation discipline additionally requires held-out split integrity (no leakage), validation-chosen thresholds, and honest reporting of negative results — see the Results section below.

Results

The figures below are the published, validation-chosen operating points from the Software Verification Report (compliance/drd/vv-report.md); thresholds are calibrated on held-out validation slices of the train split, never swept on the evaluation set.

Metric	Value	Requirement	Status
Similarity retrieval — mAP@10	0.774	REQ-F-03	Verified
Similarity retrieval — precision@10	0.822	REQ-F-03	Verified
Few-shot linear probe — macro-F1 @ 50 labels/class	0.895 ± 0.011	REQ-F-04	Verified
Few-shot linear probe — macro-F1 @ full train pool	0.920	REQ-F-04	Verified
Change detection (tile) — F1	0.510	REQ-F-05	Partially verified
Change detection (tile) — Cohen’s Kappa	0.231	REQ-F-05	Partially verified
Change detection (tile) — ROC-AUC	0.640	REQ-F-05	Partially verified

Negative / honest findings. Zero-training change detection (cosine distance between the two dates’ frozen embeddings) scores at or below chance (ROC-AUC 0.27–0.49 across every configuration) and is recorded as rejected by experiment, with the seasonality/phenology confound traced as the root cause. The frozen-foundation-model thesis survives via the cheap supervised Δembedding probe (ROC-AUC 0.640, F1 0.510), reported with its honest, validation-chosen threshold. Line/branch coverage is currently unmeasured (pytest-cov not yet wired into CI), recorded as the open gap REQ-N-05 (≥ 70 %).

Result figures

The figures below are regenerated from the public demo bundle (embeddings.parquet + probe.npz, a 2,000-tile EuroSAT subset) and illustrate the methodology end to end. They reproduce the published behaviour on the subset; the headline numbers in the table above are the full evaluation from the Verification Report (minor differences are expected on the smaller subset).

Similarity search on frozen Clay v1.5 embeddings: each query tile (left column) and its nearest neighbours retrieved by FAISS cosine search (IndexFlatIP), with EuroSAT class labels.

Per-class retrieval precision@10 on the demo-bundle held-out queries (overall P@10 = 0.80 on this subset; full-evaluation figure 0.822).

Few-shot linear-probe confusion matrix on the demo-bundle held-out split (row-normalized). Errors concentrate on the spectrally similar AnnualCrop / HerbaceousVegetation / PermanentCrop group.

Label efficiency — the foundation-model value proposition: a linear probe on frozen embeddings reaches a strong macro-F1 with only a handful of labels per class and approaches the full-train probe (mean ± std over 5 seeds).

Test report annexes

The generated machine-readable reports below are produced by the CI pipelines.

Software unit test report

Software integration test report

Raw report files