Virtual laboratories

Virtual laboratories: new opportunities for collaborative water science


Reproducibility and repeatability of experiments are the fundamental prerequisites that allow researchers to validate results and share hydrological knowledge, experience and expertise in the light of global water management problems. Virtual laboratories offer new opportunities to enable these prerequisites since they allow experimenters to share data, tools and pre-defined experimental procedures (i.e. protocols). Here we present the outcomes of a first collaborative numerical experiment undertaken by five different international research groups in a virtual laboratory to address the key issues of reproducibility and repeatability. Moving from the definition of accurate and detailed experimental protocols, a rainfall–runoff model was independently applied to 15 European catchments by the research groups and model results were collectively examined through a webbased discussion. We found that a detailed modelling protocol was crucial to ensure the comparability and reproducibility of the proposed experiment across groups. Our results suggest that sharing comprehensive and precise protocols and running the experiments within a controlled environment (e.g. virtual laboratory) is as fundamental as sharing data and tools for ensuring experiment repeatability and reproducibility across the broad scientific community and thus advancing hydrology in a more coherent way.

Science Questions

1. What factors control reproducibility in computational scientific experiments in hydrology?

2. What is the way forward to ensure reproducibility in hydrology?

Final Protocol

1) Partners

S. Ceola, E. Baratti, A. Castellarin, A. Montanari, E. Toth: University of Bologna, Italy
B. Arheimer, R. Capell, Y. Hundecha, G. Lindström: Swedish Meteorological and Hydrological Institute (SMHI), Sweden
G. Blöschl, J. Parajka, A. Viglione: Vienna University of Technology, Austria
J. Freer, D. Han, C. Hutton, T. Wagener: University of Bristol, UK
M. Hrachowitz, R. Nijzink: Delft University of Technology, the Netherlands

2) Data and Method Preparation

Set up experiment protocols: Identical implementations of the TUWmodel are distributed to the research groups, and two different protocols (i.e. Protocol 1 and 2) establishing how to perform the experiment are defined. Protocol 1 is characterised by a rigid setting, such that the researchers are required to strictly follow pre-defined rules for model calibration and validation (see Table 1), whereas the alternative Protocol 2 allows researchers more flexibility in order to explore and compare several different model calibration options (see Table 2).

Collect input data: 15 catchments (located in Sweden, Germany, Austria, Switzerland and Italy) characterised by a drainage area larger than 100 km2 with at least 10 years of daily hydrometeorological data, as lumped information on rainfall, air temperature, and runoff are considered. See MAP, catchment characteristics and input data.

Repurpose data to input files: Estimation of potential evaporation data from hourly temperature and daily potential sunshine duration by a modified Blaney-Criddle equation.

Model: TUW model: R code

3) Experiment Execution/Analysis Steps

Compute model code: perform automatic calibration and manual calibration, respectively.

Compile model results, performance metrics and parameter values into an Excel table.

Analyse differences in results between various research groups: TUWmodel performanceparameter estimationTUWmodel comparative performance

4) Result Reporting


Figure 1. Geographical location and runoff seasonality (average among the observation period listed in Table 1) (mm month−1 ) for the 15 catchments considered in the first collaborative experiment of the SWITCH-ON Virtual Water-Science Laboratory.

Figure 2. Optimal RMSE of runoff (square root of the objective function) obtained for calibration period 1 and calibration period 2 by each research group for the 15 catchments. The black bars show the range in optimal performance obtained by a single research group (BRISTOL) from 100 calibration runs initiated from different random seeds.

Figure 3. Parallel coordinate plots of the optimal parameter set estimates derived from each participant group in each of the 15 catchments for Protocol 1. Model parameters are shown on the x axis and catchments on the right-hand y axis. The parameters have been scaled to the ranges shown in Table 2.

Figure 4. Nash–Sutcliffe efficiency (NSE) estimated for model validation, obtained by the five research groups, for the 15 catchments, according to Protocols 1 and 2.

Figure 5. Flowchart of the suggested procedure to establish protocols for collaborative experiments.


Table 1. Summary of the key geographical and hydrological features for the 15 catchments considered in the first collaborative experiment of the SWITCH-ON Virtual Water-Science Laboratory.

Table 2. Main settings of Protocol 1 of the first collaborative experiment of the SWITCH-ON Virtual Water-Science Laboratory.

Table 3. Comparison among Protocol 1 and Protocol 2 settings of the first collaborative experiment of the SWITCH-ON Virtual WaterScience Laboratory.

Published Papers

Ceola, S., Arheimer, B., Baratti, E., Blöschl, G., Capell, R., Castellarin, A., Freer, J., Han, D., Hrachowitz, M., Hundecha, Y., Hutton, C., Lindström, G., Montanari, A., Nijzink, R., Parajka, J., Toth, E., Viglione, A., and Wagener, T.: Virtual laboratories: new opportunities for collaborative water science, Hydrol. Earth Syst. Sci., 19, 2101-2117, doi:10.5194/hess-19-2101-2015, 2015.

Hutton, C., Wagener, T., Freer, J., Han, D., Duffy, C., and Arheimer, B.: Computational Hydrology Is Not Reproducible, So Is It Really Science? Water Resour. Res.. 52(10):7548–7555. doi:10.1002/2016WR019285