Round Robin


The European Space Agency and the WorldWater project invited interested parties in December 2020 to participate in a Round Robin inter-comparison of inland surface water detection and monitoring algorithms using Sentinel-1, Sentinel-2 and Landsat 8 imagery.

Worldwater Round robin: First Results

One of the main goals of the the WorldWater project is to contribute to the formulation of new best practices for mapping and monitoring surface water with Earth Observation (EO) data. A particular topic is to advance the monitoring of surface water extent dynamics by taking advantage of the new enhanced capabilities of the latest generation of open and free satellite data from the European Copernicus programme. For the first time in history, the Copernicus programme provides users with access to globally and systematically acquired Synthetic Aperture Radar (SAR) data. This is a major breakthrough which can contribute to more robust monitoring in environments challenged by frequent cloud cover and periods with limited light such as high latitudes. Yet, surface water mapping with SAR data is still complicated by a number of scientific challenges (e.g., topography, wind, low-backscatter surfaces other than water). This is why the synergistic use of both optical and SAR data emerges is an interesting alternative, with the potential to take advantage of the individual sensors’ strengths, while minimizing their weaknesses.

Lately there has been some promising studies showing the strength of such a sensor fused mapping approach, yet there has been no systematic evaluation against other leading approaches for surface water mapping. Therefore, the WorldWater project organised a Round Robin exercise aiming at the inter-comparison of EO algorithms for surface water detection, using the latest generation of free and open satellite data from Sentinel-1, Sentinel-2 and Landsat 8. By comparing the robustness of different algorithms, the Round Robin is expected help us better understand the pros and cons of EO approaches for mapping and monitoring the extent of inland open waters and identifying shortfalls and areas of further research.

In total the WorldWater Round Robin was joined by 15 organizations representing a mix of research institutions, private companies, government agencies and non-governmental organizations. All participants were asked to produce monthly maps of surface water presence at 10-meter spatial resolution for 2 consecutive years and over 3 mandatory sites located in Mexico, Colombia and Zambia, and two optional sites in Gabon and Greenland, respectively. All sites were 100×100 km and the input data sets for each test site included all data acquired by Sentinel 1, Sentinel 2 and Landsat 8. Use of ancillary datasets (such as evaluation models and a priori surface water maps) was allowed, but under the condition that they are obtainable as free and open datasets.






Figure 2. Examples of surface water frequency maps over the 5 test sites.

The outputs generated by the Round Robin participants across the 3 (+2) test sites were evaluated individually and in cross-comparison using a harmonized independent reference data set. In total over 9000 samples were allocated across the 5 test sites and bimonthly in 2019 (January, March, May, July, September, November). Each sample point was assigned to be either water or non-water by a WorldWater independent validation team and shared with all participants in a fully transparent process.

Initial results confirm that high accuracies in surface water mapping can be achieved with optical and SAR data and with optical data better at capturing spatial detail while SAR data provide a better seasonal characterization. Based on these findings it is not surprising to see that sensor fused approaches outperform the single sensor approaches when looking at the performance across sites as well as through time. More, specifically we find that both supervised and unsupervised learning can provide very good results. The requirement of training data for supervised learning seems less important as global products (such as the JRC global surface water explorer) provide training data of sufficient accuracy to achieve good results.  Finally, specific steps for pre-processing and post-processing are also highly relevant for the outcome but includes many variables that are harder to quantify in terms of their individual contributions to the statistical accuracy.  Currently, the full results of the Round Robin validation are being wrapped up and with the summary results and lessons learned being targeted for a scientific journal.

Figure 3. Preliminary accuracy statistics from the WorldWater Round Robin
(OA=Overall Accuracy; UA=User’s Accuracy; PA=Producer’s Accuracy)

Ultimately, it is the hope and ambition that the WorldWater Round Robin Exercise can contribute with new knowledge and documentation which can help formulate new guidelines for decision makers and practitioners on how to adopt and use EO to better capture and report on surface water dynamics. With the Sentinel satellite constellation, data has continued to improve greatly, and combined with the advances in technical infrastructures for big data analysis, it is now within the realm of countries to implement national EO based surface water monitoring systems to support more evidence-based planning and management of water resources and for publishing and disseminating data and statistics on the SDG indicator 6.6.1. It is worth remembering that the SDG guidelines operate with a 2001–2005 baseline period, hence the historic Landsat records still hold invaluable information. Thus, there is a strong incentive and justification for the continued research into the usage of Landsat and its continuation to the Sentinels (and the increasing amount of other new sensors) for tracking long-term surface water dynamics.


The Round Robin has been organized by the WorldWater technical team under the auspice of the European Space Agency and being supported by a number of international organizations and initiatives including CNES, the European Association of Remote Sensing Companies (EARSC), the CEOS Ad hoc team on Sustainable Development Goals (CEOS SDG AHT) as well as GEO and their Earth Observations for the Sustainable Development Goals (EO4SDG) initiative.

The organizers would like to extent their thanks and appreciation to all participants in the Round Robin and who all contributed on a voluntary basis and with a shared vision to advance the field of EO based surface water detection.