Very promising results from Acpy autocalibration tool

posted in: Lake model blog | 0

Test of calibration tool for Danish lakes

In the modelling group in Denmark, we have for some time been testing the autocalibration tool developed by BB – i.e. “acpy”. Acpy is currently running for the main Prognos site in Denmark (the Lemming mesocosms), but in parallel to this work, we have also been testing the tool for lakes Ravn and Hinge, representing a deep and shallow Danish lake, respectively.

 

We have trailed the general calibration approach, which was discussed during the most recent Prognos workshop in Silkeborg (2. to 6. Oct. 2017). Here, the calibration is divided into a series of steps. Each step targets a selection of parameters that we think will have strong influence on the temporal and vertical dynamics of specific state variables. We found that for a deep stratifying lake (Lake Ravn), 5 steps resulted in overall good model performance. In general, the steps were completed in this order: physical dynamics -> oxygen dynamics -> nutrient dynamics -> phytoplankton dynamics (and higher trophic levels).

 

Calibration set up used for this test:

Warmup period (spinup years): 2006-01-01 to 2009-12-31 (4 years)

Calibration period: 2010-01-01 to 2014-12-31 (5 years)

(times series period left for validation: years 1989 to 2005)

 

The 5 individual calibration steps:

  1. Calibration of key physical parameters using observed temperature profiles for the objective function (aiming at a sharp narrowing of physical parameters, e.g. wind_factor; shf_factor, swr_factor, k_min, A, g1 and g2).
  2. Calibration of mineralization rates and initial states of sediment organic matter using observed hypolimnetic oxygen data for the objective function. The parameters with narrowed ranges from Step 1 are also included in Step 2.
  3. Calibration of nitrification and denitrification parameters and initial states of sediment nitrogen content using hypolimnetic nitrate and ammonium data for the objective function. The parameters with narrowed ranges from Step 1 and 2 are also included in Step 3.
  4. Calibration of phosphorus sorption/desorption parameters and initial states of sediment phosphorus content using hypolimnetic phosphate data for the objective function. The parameters with narrowed ranges from Step 1, 2 and 3 are also included in Step 4.
  5. Calibration of phytoplankton and higher trophic level parameters using chlorophyll a, oxygen, nitrogen and phosphorus data in the objective function (entire profiles, and not only hypolimnion data, was used in this step).

 

Methodology for each step:

For each individual step, 500-2000 simulations were typically run, after which “dotty plots” were evaluated, aiming to narrow the range of each individual model parameter (or model initial condition). After narrowing the range of one or several parameters, the calibration was re-started with the new narrower parameter ranges in the .xml file for acpy (or a new range shifted towards a higher or lower level – but ideally always constrained by limits estimated from field/lab/literature).

In practice, evaluation of dotty plots was achieved by running “acpy plot config_acpy.xml -r -100” in a separate python terminal window at discrete times that was convenient for the modeller (during an ongoing calibration run). If no clear trends in the dotty plots could be seen after a few hundred simulations, then the calibration run was allowed to continue (typically, clear trends could be seen in several parameters before reaching 1000 to 2000 simulations). If clear trends could be seen, then the parameter(s) sampling ranges were narrowed accordingly in the .xml file, the ongoing calibration run was stopped, and then re-started with the updated .xml file. A more formal way of this approach would be to run the same amount of simulations for each individual iteration (e.g. 1000, 2000 or more). This could be done by using the argument –maxfun (max. number of simulations), e.g. “acpy run config_acpy.xml –maxfun 1000”.

Note that the “-r -100” arguments used when running “acpy plot” allows you to zoom in on a range (r) at the top of the lnlikelihood space (note that lnlikelihood values will be negative), which can help with identifying more clearly any potential trends/slopes in the dotty plots (the range value could typically vary from -50 to -300 (negative) to achieve an appropriate zoom level). Note also that if you add a new model parameter to sample in the .xml file for acpy, then you should start with a fresh SQLite database (e.g., rename or delete any previous databases before running “acpy run” again).

Tip: the free DB Browser for SQLite (http://sqlitebrowser.org/) allows you to open the SQLite database produced by acpy during run. This allows you to quickly sort the “lnlikelihood” column and thereby monitor the status of the best performing simulations (e.g. is the best lnlikelihood still increasing over time/simulations?).

Typically, 5-15 iterations (each comprising about 500-2000 simulations) were completed for each of the five calibration steps, after which the dotty plots would typically no longer exhibit clear trends/slopes for the individual model parameters. Most iterations were done in the final step (e.g., 15 or more).

 

Initial results:

As we have worked with Lake Ravn in previous modelling projects, based mainly on the DYRESM-CAEDYM model, we can readily compare the new calibration results (using GOTM-FABM-PCLake and the acpy tool) with that achieved earlier. For all the state variables tested, the new GOTM-FABM-PCLake generally performed better than our previous efforts. This applies to simulated temperature and oxygen profiles, nutrients (inorganic fractions as well as totals) and total chlorophyll a. In terms of statistical performance, simulated chlorophyll a showed the weakest performance, yet the results were still very good compared to earlier work (see figure below). Calibration is still ongoing, but the acpy tool has saved us a lot of time in achieving a model with publication-worthy performance. We aim to eventually produce a manuscript on the work on Lake Ravn, which outlines the calibration approach and our experiences with the individual steps of the calibration.

 

Plot generated using WET (wet.au.dk)