ESA PROBA-1 CHRIS Reprocessing: Cloud and Cloud-Shadow Detection Module

Open in Colab

Cloud and cloud-shadow segmentation for CHRIS/PROBA-1 imagery, with three classes: clear, cloud and shadow. It returns a label map for the whole scene.

A CHRIS file is not four bands. It holds dozens of narrow bands, in an order that changes with the acquisition mode, and their centres rarely land on the wide Sentinel-2 red, green and blue. The model always takes the same four inputs, close to Sentinel-2 B4, B3, B2 and B8, so for every mode the loader first groups the CHRIS bands into those four channels, weighting them by how well their spectral response matches the Sentinel-2 band. The weights are in bands.json, one entry per mode. Nothing else has to be done by hand: predict_chris reads the raw cube, builds the four channels and segments the scene.

pip install mlstac pytorch-lightning segmentation-models-pytorch timm rasterio
import mlstac

model = mlstac.load("https://huggingface.co/isp-uv-es/CHRIS-PROBA1/resolve/main/mlm.json")
local = model.download("CHRIS-PROBA1")
net = local.compiled_model(device="cuda")

mask = local.module.predict_chris("scene_TOA.tif", model=net, mode_n=3, source="toa")
# 0 = clear, 1 = cloud, 2 = shadow, 255 = nodata

example

Accuracy

On 55 hand-painted CHRIS scenes, split by flight and never seen in training, each scene going through the full inference path in load.py:

Metric Value Aggregation
mIoU 0.708 mean per scene
IoU clear / cloud / shadow 0.898 / 0.831 / 0.652 pixels pooled
Macro F1 0.881 pixels pooled
Cohen's kappa 0.854 pixels pooled
Accuracy 0.921 pixels pooled

The first row averages scenes; the rest pool every valid pixel. They are not interchangeable: the pooled mIoU is 0.793, the per-scene mean 0.708.

Per mode, mean mIoU and number of scenes: mode 1 0.784 (12), mode 2 0.697 (5), mode 3 0.699 (12), mode 4 0.576 (5), mode 5 0.658 (3), mode 20 0.711 (18). Modes 2, 4 and 5 have five scenes or fewer, so those numbers are indicative only.

metrics

How it works

Three U-Nets, averaged:

Checkpoint Encoder
checkpoints/convnext_a.ckpt ConvNeXtV2-nano
checkpoints/convnext_b.ckpt ConvNeXtV2-nano
checkpoints/regnety.ckpt RegNetY-004

The two ConvNeXt share architecture and bands; they differ in the labels they learned from, so their errors do not line up.

Inference walks the scene in 640 px windows with 320 px overlap, snaps the last window to the edge so nothing is padded, normalises each window per band with a z-score, averages the softmax of the three checkpoints and blends the windows with a gradient mask. Around 0.3 s per 748 x 744 scene on an RTX A5500 with the weights loaded, 4 s on CPU.

From CHRIS bands to the four inputs

CHRIS is a pushbroom spectrometer: one mode records 18 bands, another 62, and the order changes with the mode. The model was trained on four wide channels, so the conversion is a weighted mean of the native bands that fall close to each Sentinel-2 band. In the figure, every panel is one mode, every lane is one input, the wide curve is the Sentinel-2 response and each block is a CHRIS band that feeds it.

Per mode, the CHRIS bands that build each of the four inputs

Mode 3 records 18 bands and the four inputs use 9 of them:

Input CHRIS bands (index, weight)
B4 red 7 (0.64) + 8 (0.36)
B3 green 4 (0.59) + 5 (0.41)
B2 blue 2 (0.93) + 3 (0.07)
B8 NIR 14 (0.41) + 15 (0.45) + 16 (0.14)

The weight is not a guess. For every CHRIS band the response is compared with the Sentinel-2 response of that channel and the similarity is normalised inside the group, so a band that sits under the Sentinel-2 curve carries most of the input. Several comparison kernels (plain mean, box, raised cosine, bell shaped) were tested against real Sentinel-2 radiance and landed within 0.0002 of each other in correlation, so what the kernel is matters less than which bands it keeps.

Cosine similarity between the CHRIS band responses and the Sentinel-2 curve, mode 3

Two things follow. A band far from every Sentinel-2 centre is dropped, so a mode with 62 bands does not feed 62 inputs. And when several CHRIS bands sit inside one Sentinel-2 band, they are averaged, which is what gives the same four inputs from every mode. Every checkpoint, whatever its mode, reads those same four channels; the mode only changes which native bands are averaged behind them. If you already have the four channels, skip this and call predict_large directly.

The same rule covers all 13 Sentinel-2 bands, not only the four used here, so a model with more inputs can be built from it. In mode 3, adding B5 would take bands 9 (0.65), 10 (0.33) and 11 (0.02). The full set is in bands_full.json; the four channels this release expects are in bands.json.

Inputs

Each mode has its own set of CHRIS bands and weights. Mode 3, for example, builds B4 from two adjacent bands, while mode 1 needs four bands for B4 and twelve for B8. To see what a mode uses:

local.module.band_table()["3"]

The file you pass can be in three radiometric states:

source What the file holds
"toa" top-of-atmosphere reflectance, scaled by 10000
"rad" radiance as the sensor recorded it
"dn" raw digital numbers, uncalibrated

All three load, because the per-window z-score removes the global scale. Values are divided by 10000 and clipped at 2.0 before that, so extreme radiance can saturate. TOA and radiance are the accurate ones; DN is a fallback. Mode 20 is passed as mode_n=20 and is normally delivered as radiance.

The input is a GeoTIFF with the CHRIS bands in their native order, counted from

  1. Pixels with no data must be 0. All the examples are near-nadir; off-nadir acquisitions were not tested.

Modes 1, 2, 3, 4, 5 and 20 are supported. Mode 1 is binned to 372 x 374 by the instrument, so its window shrinks to 352 px; the others are 744 px wide, with 748 rows in most acquisitions and up to 1024 in some mode 2 products.

With your own four-channel stack:

mask = local.module.predict_large(rgbn, model=net)   # rgbn: (4, H, W) float

Per-pixel confidence:

probs = local.module.predict_chris("scene_TOA.tif", model=net, mode_n=3,
                                   source="toa", return_probs=True)

Walkthrough with the six modes, a GPU against CPU timing and a batch example: notebook.ipynb. The Colab badge at the top opens an editable copy in Google Drive.

modes

Data

  • Pretraining: CloudSEN12, 50,247 patches resampled to 640 x 640 at 10 m with 13 spectral bands, labelled clear / cloud / shadow. Eleven band combinations were trained with ConvNeXtV2-nano and RegNetY-004. Those with four bands or more land within 0.008 mIoU of each other in our runs, so the four-band one was kept. It is the cheapest and the only one every CHRIS mode can build.
  • Fine-tuning: 130 hand-painted CHRIS scenes, each sampled 15 times with random crops, plus 1,611 automatic masks over scenes nobody painted. A second round retrained on the model's own predictions, keeping a pixel only where the prediction was confident and stable.
  • Test: 55 painted scenes, split by flight, never by scene, so two angles of the same acquisition never sit on both sides.

What it is not

  • Shadow is the weak class (IoU 0.652). CHRIS covers 406 to 1005 nm, with no SWIR and no thermal band, so a shadow and a dark surface look alike. Expect misses over water and dense vegetation, where sun geometry and a DEM separate the two better than spectra do.
  • No thin-cloud class. Thick and thin are merged into cloud.
  • Differences under 0.03 mIoU are noise with 55 test scenes.
  • Mode 2 after 2015 is unusable: those products store a gain setting of -1, so the raw calibrated imagery (RCI) is not calibrated and the digital numbers are meaningless.
  • The three checkpoint files carry the optimizer state, which is why they weigh more than the networks.

Files

load.py                 mlstac loader
mlm.json                model card, STAC/mlm format
bands.json              CHRIS to RGBN table per acquisition mode
bands_full.json         same rule for the 13 Sentinel-2 bands
checkpoints/            the three networks
examples/mode_<n>/      one scene per mode (mode_20 for 20), mask and prediction
figures/                metrics, example, gallery
notebook.ipynb          Colab demo
metrics.json            per-class and per-mode numbers

MIT licence.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support