Calibrating and validating a combined accumulation and mobilisation model for water distribution system discolouration using particle swarm optimisation

Abstract

A discolouration concept is proposed describing simultaneous pipe wall material accumulation and mobilisation processes that define discolouration in drinking water distribution systems, one of the biggest causes of customer dissatisfaction. Validation of these mathematical forms is presented. The model formulation was shown to maintain the mobilisation functionality of previous validated shear-stress-dependant modelling tool, but requiring only two empirical parameters. Two distinct operational datasets are then analysed and robust empirical model parameter calibration is obtained utilising a refined particle swarm optimisation technique. The model is shown to make usefully accurate simulations for flow mediated events, providing evidence of predictive capabilities. The combined tracking of accumulation and mobilisation behaviour enables assessment of the current and future discolouration risk posed by any pipe irrespective of age or material, allowing pro-active, risk based planning and prioritisation of maintenance interventions to protect the quality of delivered water.

Introduction

Discolouration material is acknowledged to accumulate continually and ubiquitously across all pipe surfaces in drinking water distribution systems (DWDS) (van Summeren and Blokker 2017, Vreeburg et al. 2008). As a result, any maintenance intervention for managing discolouration risk, such as disruptive invasive techniques or in-operation flow conditioning as developed in the UK to periodically mobilise accumulated material using controlled increases in system shear stress (Husband and Boxall 2016), can only be considered a temporary strategy. Of operational significance therefore is the rate at which particulate matter re-accumulates on the pipe walls, increasing the discolouration risk as the asset condition deteriorates and the frequency at which interventions are required. Models describing the mobilisation of material from pipe walls due to elevated hydraulic forces exist (Vreeburg and Boxall, 2007) and have been verified in the field for distribution pipes and trunk mains (Husband and Boxall, 2010, Husband and Boxall 2016). However, models of material accumulation do not reflect process knowledge obtained from operational experience and supported by extensive field and laboratory studies.

Discolouration risk is difficult to measure directly due to its origin as material on the inside of buried pipe infrastructure. Turbidity and flow data characterising the discolouration response can be collected, however due to the short term sporadic and spatially variable nature of ‘natural’ discolouration events this can be complex (Furnass et al. 2013). However it is possible to collect discolouration data around planned events that disturb pipe accumulated material, although these have inherent risks to water quality and customer perceptions. Water companies therefore need a means for assessing the current discolouration risk not as a complex, typically noisy flow and turbidity time-series, but as a simple, comparable metric. In addition, strategic planning requires a way of assessing how discolouration risk changes in the short, medium and long term under multiple hydraulic scenarios.

With continuous material accumulation highlighting the restricted short-term value from one-off strategies such as pipe renewal or invasive cleaning, maintenance programmes of regular managed flow increases offer a potential for long-term low cost risk management. However, hydraulics and water quality in DWDS are difficult to characterise as they are topographically complex bio-reactors with large surface areas, predominantly access restricted and with limited monitoring opportunities. Without pro-active discolouration risk management, any increase in flow resulting from planned or unplanned events can mobilise accumulated material. As this material represents long term accumulation of low level background components, the hazards to water quality extend beyond simply aesthetic discolouration and can include elevated concentrations such as for iron, manganese or lead, disinfection by-products and microbial biofilm content including potential pathogens. Operational flexibility is however paramount for the management of distribution systems and in particular trunk mains, where failure consequences can be extensive. Water providers therefore need to be able to conduct scheduled maintenance, such as to treatment works, pumping stations and service reservoirs, and also react promptly when such assets fail. These need to be conducted while minimising any negative impact on water quality. It is therefore vital that discolouration processes are well characterised to aid with their management. Efforts to achieve this have however been historically impeded by the heterogeneity of DWDS and by the practical requirements of pipe-specific discolouration studies.

It is difficult to characterise well the biofilms/layers that accumulate on the walls of DWDS without disturbing them (Douterelo et al. 2016, Husband et al. 2016). A model capable of describing material mobilisation and accumulation processes, plus the contributing factors, would equip water providers with the means to better schedule and design both flow conditioning and general maintenance programmes. This would reduce the risk of customers being supplied with discoloured or poor quality water and by understanding return periods would also facilitate whole life costing of those programmes. This paper presents a model that combines accumulation and mobilisation processes and its application to field experimental data. Implementation in a software Application Program Interface (API) allowed validation by calibration to real world data.

Background

The detection of discoloured water at customers’ taps has been shown to be predominately the result of a change in hydraulic conditions within the upstream DWDS mobilising wall bound particulate material (Husband and Boxall, 2010), as shown in Fig. 1. Proactive, operational strategies monitoring the turbidity response to managed flow events have evidenced this as a cost-effective means for reducing the likelihood of discolouration events (Husband and Boxall, 2016). In addition, discolouration modelling has been validated as capable of predicting the turbidity response, due to material mobilisation, when preceding peak flows are exceeded.

Fig. 1

The interior of a ductile iron main that was drained down to repair a leak. Note the thin (particulate) material coating the complete pipe wall, as well as larger more stable corrosion tubercles

Over time continual particulate material flux in the bulk water causes material accumulation on the internal surfaces of DWDS in cohesive layers (Husband and Boxall 2011). The rate of accumulation on pipe walls is a complex function driven by multiple processes including corrosion, adhesion and microbial growth (Abe et al. 2012, Matthieu et al. 2014, Mounce et al., 2016). With a heterogeneous and complex composition, wall-bound discolouration material does not have a single binding shear strength. Most pipes in operational networks do not pose a discolouration risk on a typical daily basis, indicating that accumulated material has a strength versus quantity profile with the lower bound of the strength range defined by the prevailing shear stress i.e. discolouration material is conditioned by the usual pipe hydraulics. The strength profile of wall-bound material can be studied by increasing the shear stress incrementally above the ‘background’ and measuring how much material (the turbidity response) is mobilised (Husband et al. 2008). One way to extend this investigation to examine how material then accumulates over different strength ranges is to then repeat this exercise after a defined period (Husband and Boxall, 2011).

Modelling provides a way of encoding system state information and transaction rules in an abstract form that allows for extrapolation. For example, the above understanding of material mobilisation may be captured by describing mobilisation of wall-bound material occurring when the applied shear stress, τa, exceeds the shear strength, τ, of bound material. This is the basis of the empirical Prediction of Discolouration in Distribution Systems (PODDS) discolouration model (Boxall et al., 2001). This model does not attempt to model corrosion processes, biofilm development, suspended iron flocs or chunks of biofilm directly; instead, material mobilisation is considered to be the mechanism by which turbidity causing, homogeneous material can move between being bound to the pipe wall and being a wash load suspended in the bulk water. The PODDS model estimates the amount of material (bound to the pipe wall) by assuming an equilibrium state with the normal daily hydraulic conditions and then simulates the turbidity response that would result from an excess shear stress event. PODDS model parameters are empirical, estimated through calibration to measured data. Flow trials (flushing of distribution pipework or raising flows in larger diameter trunk mains) are used to generate the turbidity, flow and headloss data required for such calibration. Headlosses can either be estimated using a pipe roughness or measured directly during trials, facilitating concurrent hydraulic network model calibration. PODDS model parameters have demonstrated transferability between sites with analysis showing them sensitive to pipe diameter, pipe material and source water (Boxall and Saul, 2005).

While the PODDS model has been validated to represent mobilisation behaviour, it does not describe the material accumulation processes observed in both field and laboratory studies. In Furnass et al. (2014) a new model, the Variable Condition Discolouration Model (VCDM), was proposed to overcome this shortcoming. It is postulated that wall-bound material with τ ≤ τa is eroded whilst simultaneously material for which τa < τ < τmax accumulates (where τmax is the maximum possible τ with which material binds to the wall). The model therefore tracks the relative amount of discolouration material bound to the pipe wall over time at each of a number of shear strengths. A shear-strength-variable yet time-invariant mobilisation model (see Fig. 2) is combined with the accumulation model as a) it allows for non-serial layer mobilisation so can account for spatial heterogeneity in material composition; and b) it can ensure that the mobilisation rate is a function of excess shear stress. The PODDS model captures mobilisation as a process progressing in series from strong to weak layer, in series. Whereas observation of wall bound material is shown to be heterogeneous at the micro scale (Fish et al. 2016 and Fish et al. 2017), with different strength material exposed simultaneously. Figure 2 attempts to capture this, showing how the distribution of amount of material (φ) across different strengths (τ) would be simultaneously mobilised, but at decreasing amounts, due to each time step of an imposed excess shear stress.

Fig. 2

A mobilisation model where all shear strengths erode simultaneously at a rate that is constant with time for each shear strength but varies between shear strengths, with the weaker material eroding more quickly as driving force (τa-τ) is greater

Modelling the accumulation rate could be achieved using a process model, taking account of many factors such as corrosion, temperature and biofilm growth. This however would require a sufficiently-detailed understanding of the inter-dependencies (a set of physical, chemical and biological reactions per distinct environment, with the stoichiometry and kinetics/equilibria being characterised for each biological and chemical reaction), hence likely to be difficult and impractical. For the purposes of this work, material accumulation is considered to be a simple, empirical rate that is a scalar and time-invariant. As further data and process knowledge becomes available, this rate-based model can be refined through decomposition into empirical and process-aware sub-models.

The output from the mobilisation and accumulation model is a rate of material release from the pipe wall into the bulk flow. Together with an understanding of system hydraulics, this can then be fed into existing water quality models to account for advection and mixing and to calculate turbidity at any downstream location. A series of pipe lengths with differing hydraulic or asset characteristics can collectively be modelled in a ‘daisy-chained’ fashion: the turbidity predictions from modelling the most-upstream pipe are supplied as input when modelling the change of state and turbidity for the next pipe downstream, and the output from that sub-model is used as input when considering the next pipe downstream etc. (see Fig. 3).

Fig. 3

Discolouration material release and propagation can be modelled in heterogeneous pipes by supplying the output of each sub-model as an input to the downstream sub-model

The VCDM model has three parameters, a relative mobilisation or erosion rate factor βe, an accumulation or regeneration of material rate βr, and a scaling factor α for converting from relative to absolute quantities of discolouration material. The wall state boundary condition for t = 0 is the relative material quantity per tracked shear strength, which is a non-trivial monotonically-increasing function of τ. The material condition function φ (τ,t) is the relative amount of discolouration material bound to the pipe wall with shear strength τ at time t. τ is a vector property, in that it is used to track the range of material strengths at every position on a pipe, and φ (τ,t) ∈ [0,1] where 0 corresponds to complete depletion (no material) and 1 to maximum accumulation.

VCDM detail and mathematical formulations can be found in Furnass et al. (2014) and Furnass (2015). In these publications formulation verification was conducted with artificial data and limited data from operational networks and using manual calibration. The proposed model however still requires validation, primarily to explore the error in predictions made using calibrated models and continuous flow data. The proposed model is only of greater value than PODDS if both mobilisation and accumulation mechanisms can be shown to be valid. The proposed model of material mobilisation differs from the validated PODDS mobilisation mechanism so it must be independently validated.

The aim of the research reported is to calibrate and validate a formulation that attempts to describe the continual processes of material accumulation and mobilisation (due to hydraulic changes) within DWDS and hence inform asset management for the control of discolouration. Specifically, the paper presents information on:

• Confirmation that the proposed formulation, with only two mobilisation parameters, retains functionality to describe material mobilisation as currently described by the PODDS model, requiring three parameters.

• Implementation of the proposed formulation within a Python environment, in particular integration with optimisation algorithms to enable efficient and repeatable calibration and automated systems to identify periods of interest from long time series data.

• To provide evidence of the ability of the model to describe the long term behaviour of both continual material accumulation and mobilisation within a trunk main through calibration, and to provide initial validation of predictive simulations once calibrated.

Methods

Validation of the discolouration mobilisation model can be achieved using field data obtained during the flushing of pipes. Flushing durations are sufficiently short relative to the time required for material accumulation, irrespective of if this is within larger trunk mains or smaller diameter District Meter Area (DMA) pipes (Husband and Boxall, 2016), that the accumulation can be assumed to be negligible over the duration of a flush. If flushing is repeated at suitable intervals the processes of accumulation may also be investigated. The relationships between the quantities explicitly referenced by the proposed VCDM are shown in Fig. 4. Temperature and fluid density are not explicitly included in the model as the impact of variation in those factors in DWDS hydraulics is considered negligible. Temperature / seasonal variations are known to influence material accumulation processes, but are not captured in the datasets available here.

Fig. 4

Sequence calculations implemented to perform turbidity simulations. Darker shading indicates VCDM aspects leading to downstream (ds) node observed turbidity (T). Lighter shading indicates inputs and calculations that are standard in 1D network modelling software. ks: pipe roughness; S₀: hydraulic gradient; τ_a: shear stress; t: time; f: Darcy Weisbach friction factor; Re: Reynolds number; Q: flow; D: internal pipe diameter; ρ: fluid density; β_e, β_r and α: VCDM parameters; ϕ (τ, t): relative amount of material on the pipe wall with strength τ at time t; ϕ (τ, t₀): relative amount of material on the pipe wall with strength τ at time t₀; dN (t) /dt: rate of material release per unit area of pipe wall; L: pipe length; T_us: turbidity at the upstream node; T_ds_pred: predicted turbidity at the downstream node. indicates a model input. indicates a model process. Light shading indicates hydraulic model inputs and processes, while dark shading indicates VCDM specific ones

The first stage in the confirmation of the formulation was the selection and implementation of search and optimisation routines to enable automated calibration. Key functionality sought was comprehensive exploration of prescribed parameter space, efficiency and repeatability. Automated functionality was also developed to enable identification and tagging of periods of interest in long time series flow and turbidity data. This helped enable visual inspection to focus on periods of interest, and could potentially be used to focus calibration by biasing fitting to these periods. The second stage was to use the automated calibration routines to confirm the mobilisation functionality of the new formulation. This is presented for four short duration DMA flushing responses as first reported in Boxall and Saul (2005) and Husband et al. (2010). The ability of the new model to simulate long term continual accumulation and mobilisation was then assessed, requiring long time series flow and turbidity data. This is presented here as calibration to two different long term data sets. The second data set was sufficient to enable segregation into training and testing data allowing validation. Sufficient was judged to be a time series that contained a sequence of at least 5 clear mobilisation events with sufficient periods between for measurable (in terms of turbidity response when mobilised) accumulation to have occurred between them for training, and then a further 3 events for testing. The periods and magnitude necessary for this are system and events specific.

Automated event identification

Identifying the times when turbidity events occurred or are most likely to have occurred from continuous turbidity and shear stress datasets covering extended time periods is not straightforward and automation is therefore highly desirable. The motivation for identifying the times of likely events is that the data over these periods provide a response that is critical to facilitate calibration. Model validation can therefore be tailored to better qualify/quantify the ability of the VCDM to characterise the processes of interest by focussing on these disjointed subsets of the time-series dataset. Turbidity events due to mobilisation within a pipe are most likely when the pipe shear stress exceeds recent maximum. The set of likely turbidity events of a given dataset was identified as follows:

• 1. For each timestamp t, mark t as being an event time if τ_a (t) exceeds the maximum of the previous t_win seconds;

• 2. (Optional) All identified events that correspond to just a single timestamp should be classified as not being events as they could just be due to measurement error;

• 3. For all remaining events, consider all timestamps within n_turnovers×t_t after each event to be part of that event, thus allowing for discolouration material propagation following the flow increase.

Particle swarm optimisation

Evolutionary optimisation techniques such as genetic algorithms (Holland, 1992) mimic the competition inherent in natural selection to iteratively move towards an optimal candidate solution. Swarm optimisation methods are also stochastic but use collaboration rather than competition between candidate solutions to converge on the global optima. One of the more commonly-used techniques is particle swarm optimisation (PSO) (Kennedy et al., 2001).

In PSO, each dimension corresponds to an optimisation problem variable (model parameter). For the bird flocking problem, the objective function that needs to be minimised could take a set of coordinates in three-dimensional space as its three parameters and return the reciprocal of the concentration of flies at those coordinates. Initially a number of birds or particles (typically between 20 and 50 (Eberhart and Shi, 2001)) are assigned random positions and velocities within the problem space. The coordinates of each bird represent a candidate solution. The birds then move about this problem space over a number of timesteps. At each timestep each bird calculates a new velocity based on three factors (Floreano and Mattiussi, 2008):

• Inertia: its current velocity vector

• Nostalgia: the problem space position in the bird’s memory since t = 0 where the fitness value was greatest;

• Societal influence: the position of the neighbouring bird with the best fitness at the current timestep (birds may use calls to communicate their current fitness values to each other).

PSO has been used to design distribution networks with optimal topologies and pipe diameters (Suribabu and Neelakantan, 2006), develop optimal reservoir management strategies (Reddy and Nagesh Kumar, 2007), learn the optimal parameters of a lake water quality model (Campbell and Phinn, 2009), parameterise hydrological and hydrogeological flow models (Zambrano-Bigiarini and Rojas, 2013) and learn formula parameters for solute mixing at DWDS junctions (Yu et al., 2014).

Two advantages of PSO over evolutionary optimisation approaches such as genetic algorithms are that PSO is purportedly more efficient than genetic algorithms and the key algorithmic components can be encapsulated in very little code (Kennedy et al., 2001). A possible disadvantage of PSO is that it is not naturally suited to optimising problems involving categorical variables (Floreano and Mattiussi, 2008). However, with no categorical input parameters, as with VCDM, PSO is desirable due to its simplicity and efficiency of design, implementation and execution.

In the present work, model parameters were fitted using a robust PSO metaheuristic, as sensitivity analysis showed that there were many similar ways in which model parameters and measurable/calculable properties, such as flow and pipe diameter, could influence the shape and scale of time-series model outputs (Furnass 2015). The PSO objective function was the error between time-series turbidity observations and predictions, optionally calculated only at times when erosion events were likely. The dissimilarity metric used in the objective function is the integral of squared errors (ISE) and was chosen as it is moderately sensitive to the magnitude of transformations, is very insensitive to noise, is simple to calculate (Eq. 1) and is dimensional.

$$ ISE\kern0.5em \left({T}_{obs},\kern0.5em {T}_{pred}\right)\kern0.5em =\sum {\left({T}_{obs}\kern0.5em (t)\hbox{--} {T}_{pred}\kern0.5em (t)\right)}^2\kern0.5em \forall t\kern0.5em \in \kern0.5em {t}_{compare} $$

(1)

where tcompare is the set of times at which errors are to be calculated (which do not need to be contiguous).

Implementation

For the purposes of validating the proposed model for a single pipe, a bespoke software library was developed in Python (Lutz, 2011). Python is suitable for this task as it is interpreted and dynamically typed, facilitating rapid prototyping, data exploration and experimentation, yet can interface with statically typed, compiled extension modules for expediting computationally-intensive tasks (e.g. via Cython (Behnel et al., 2011)). Python was thus considered to offer the best balance of efficiency and ease of development and the greatest flexibility with respect to model fitting and sensitivity analysis.

The freeware pyvcdm Python package implementation provides VCDM coding and development functionality with which discolouration material release models can be instantiated then executed for a given time-series before modelling advection and mixing for that process i.e. the wall state change and material release code is largely decoupled from the advection and mixing code. This decoupling facilitates VCDM coding as an add-on into existing hydraulic software.

An implementation of PSO in Python, pyshoal was developed to provide the features and satisfy the model requirements and provides a simple API. Once instantiated, an optimisation run continues until convergence tolerances are satisfied for a number of iterations or a maximum number of iterations are reached. The outputs following a run are the best objective function parameter values found during the run, the value of the objective function given those parameter values and the number of iterations. The software allows for the optimisation of an arbitrary objective function (which can take any number of parameters) and can execute objective functions in parallel using multiple processes for reducing the execution time of a single optimisation run.

The calibration of model parameters can be formulated as an optimisation problem to be solved in an automated fashion:

• The objective function to be minimised is an appropriately sensitive scalar measure of the dissimilarity (the integral of squared errors was used as the metric) between turbidity observations and model instance turbidity predictions over the calibration period;

• The parameters to be optimised are VCDM model parameters βe, α and βr (unless one is confident that the duration of the simulation period is negligible compared to the time required for full regeneration in which case only βe and α);

• The optimisation constraints are independent parameter bounds for βe, α and βr: all three parameters must be positive and finite.

The approach used for calibrating the model parameters is to fit all three parameters simultaneously given static and time-series observations of bulk water properties, a PSO configuration (including parameter bounds) and known initial wall state boundary conditions (ϕ (τ, t0)). This approach, as illustrated in Fig. 5, is referred to as FIT_SIMULT. Options for fitting/estimating initial wall state boundary conditions are discussed in Furnass (2015).

Fig. 5

A data flow chart showing how all three VCDM parameters can be fitted to data simultaneously using PSO (FIT_SIMULT method)

The PSO configuration options used when fitting the proposed model to data are specified in Table 1 (excluding the specific parameter space bounds and convergence tolerance values as these are more case-specific). As PSO is a non-deterministic metaheuristic that is likely but not guaranteed to find the global optimum, each of this set of fitting tests was attempted up to three times to achieve a satisfactory fit and confirm that an optimal, repeatable solution was found.

Table 1 Generic PSO configuration used when fitting the proposed model to data

Fitting VCDM mobilisation to four DMA flushing events

The mobilisation and accumulation processes operate over dramatically different time frames. Mobilisation and associated change in pipe wall bound material and turbidity response is in terms of minutes, whereas accumulation processes and associated change in pipe bound material layer state are in terms of weeks and months. Hence by considering only short time period mobilisation events the effects of accumulation, and the associated model functionality, can be ignored. Therefore the mobilisation functionality can effectively be validated independently of the accumulation function.

The VCDM was fitted to four independent flushing events to validate the mobilisation functionality across a range of pipe materials, pipe roughness and source waters. The four selected flushes were from DMA’s around the UK and were first presented in Boxall and Saul (2005) and later in Husband et al. (2010) with corresponding PODDS model calibrations. The four flush examples (PODDS-CI1, PODDS-CI2, PODDS_PE1, PODDS-PE2), shown in Fig. 6, were used to explore the accuracy and precision (repeatability) with which the VCDM can be fitted and the quality of model fit (using a relative dissimilarity metric) compared to the corresponding PODDS calibrations.

Fig. 6

Selected DMA flushing data showing imposed flow rate (dashed line), measured turbidity response (crosses) and PODDS model simulation results (solid line)

The VCDM was calibrated using each of the DMA flushing datasets by using the FIT_SIMULT method to fit the mobilisation parameters β_e and α. The βr parameter was set to 0 s⁻¹, simplifying the fitting process as for these short duration simulations the accumulation effects would be negligible. Prior to flushing, according to company records, the pipes effected had only experienced a regular daily flow pattern for the preceding years so could be considered as ‘previously undisturbed’. As a result, the initial wall state boundary conditions were fitted using a single cusp approach: the relative amount of material at the start of the simulation was polarised around a single shear strength τ_c, which, like β_e and α, was fitted by FIT_SIMULT. Here τ_c is conceptually similar to the initial layer strength term in the PODDS model which was also derived from the preceding peak flow from the daily pattern. A fourth quantity was fitted by FIT_SIMULT, this being the maximum shear strength, τ_max, with which material can be bound to the walls of pipes. In operational terms τ_max is effectively a ‘self-cleaning’ boundary, that is the strongest force with which erodible material can accumulate at the pipe wall; for pipes experiencing daily forces in excess, material accumulation is mitigated. As with the prior PODDS simulations, τ_max was only fitted for the PE pipes as no evidence of a maximum shear strength has been found in CI pipes (Husband et al. 2010).

Fitting the VCDM to trunk main (TM-NR)

The most informative method for validating the accumulation mechanism and evidencing its practical value is to assess whether it can represent a series of turbidity responses identified as due to hydraulic mobilisation during an extended time period from a relatively homogeneous pipe length. In this first case study, the model was fitted to a 22-month dataset with a number of shear stress events and where the spacing between events was such that significant accumulation could occur. As part of an integrated network these events where in response to network demands and coordinated from a central control room. Trunk main TM-NR is a 15 km large-diameter (800 mm), lined steel trunk main that is situated in the north-east of England supplied with surface water. The main runs between a break pressure tank (BPT) and a service reservoir (SR) see Fig. 7.

Fig. 7

A network schematic showing the location of trunk main TM-NR in relation to the supplying water treatment works (WTW) and various service reservoirs (SRs) and break pressure tanks (BPTs). The schematic also shows the locations of continuous flow (Q) and turbidity (NTU) monitoring equipment

After pre-processing to remove unrealistic flow values such as during flow meter maintenance and standardising to SI units, the dataset from trunk main TM-NR was used for calibrating and validating the VCDM is shown in Fig. 8.

Fig. 8

Flow (top plot) and turbidity from trunk main TM-NR used for calibrating and validating the VCDM. Note that the upper bound of the turbidity y-axis scale has been set to 1 NTU to show the variation between mobilisation events of the turbidity of the WTW finals (Twtw_obs) and the turbidity at the downstream end of TM-NR (Tds_obs)

The automated event detection method was applied to the TM-NR dataset and identified increased shear stress using a time window of fourteen days and the turbidity response highlighted for a propagation duration of five pipe turnovers. Figure 9 shows a temporal zoom of Fig. 8 with the identified turbidity responses due to increases in shear stress highlighted in red. Using this technique, a total 39 events were detected in this dataset. The three model parameters (β_e, α and like β_r) were fitted using the FIT_SIMULT method. Unlike calibration of the mobilisation process for the flushing operations with the initial wall state boundary conditions (ϕ (τ, t₀)) polarised around a defined peak daily flow, the more variable trunk main flow history can generate a range of layer states. Fitting options are discussed in Furnass (2015), whilst in practice the simplest method as applied here sets initial wall state material as greater than would be expected with the opening flow data. This is rapidly mobilised (β_e> > β_r) with the initial week’s simulation results accepted as poor while a realistic wall state description is defined.

Fig. 9

Correlation between shear stress increases identified using 14 day time window (top plot) and subsequent 5-turnover turbidity response (red crosses superimposed on downstream turbidity, bottom plot)

Fitting the VCDM to trunk main (TM-WB)

The second trunk main case study, like the study of TM-NR, made use of a multi-year dataset. Unlike TM1 however there is little demand driven variation, as a result planned interventions featuring managed flow trials were instigated to investigate the practical application of the concepts described by the VCDM. Also unlike TM-NR which is a surface water supplied trunk main, TM-WB is supplied with treated ground water and runs between two SRs in the south-west of England. The reach referred to as TM-WB is a series of mains of varying materials (AC and unlined DI), with changing diameters and flows due to there being take-offs along its length (see Fig. 10).

Fig. 10

A network schematic showing attributes of trunk main TM-WB; AC or unlined DI, length (L) per section, internal diameter (D) and the roughness (k_s; values taken from a calibrated hydraulic model). Take-offs along the length with range of demand flows per node, the type of flow profile (domestic or stepped) and the duration over which the flow profile patterns repeat

Operators of this network wished to be able to support resilience flows should additional demand be required in the event of a WTW fail, SR fail or major burst, without there being a negative impact on water quality. To achieve this an increase from the initial (pre October 2009) operating maximum of 1100 m³ day^− 1 to 2800 m³ day^− 1 was identified. A number of previous discolouration events associated to hydraulic triggers in this trunk main had indicated that there was a significant risk of discolouration. The planned and monitored trials reported herein covered a period were therefore used to investigate the behaviour of TM-WB to flow increases and to determine if an in-situ hydraulic strategy could be applied to facilitate this resilience work.

From June 2008 investigations commenced with concurrent flow and turbidity monitoring trials. During flow increase trials that were predicted to mobilise significant quantities of material the downstream reservoir was isolated for later discharge to safeguard the downstream network. The data set with large flow changes due to resilience targeting would help confirm the mobilisation functionality of VCDM whilst the varying periodic returns was ideal to validate the accumulation functionality. Once resilience had been achieved it was planned that the flow in the main would be periodically elevated back up to the target level to maintain resilience by removing any fresh material accumulation. To facilitate this an actuated valve and permanent turbidity meter were later installed on the inlet to the downstream SR for accurate and remote flow control and future flow conditioning. Turbidity observations were logged at 15 min via Supervisory Control And Data Acquisition (SCADA) from the date of commissioning. Some of the key events in the history of TM-WB regarding discolouration risk management are shown in Table 2.

Table 2 Key events in the history of trunk main TM-WB. Note that during the flow trials turbidity was often recorded at a high temporal resolution but the flow measured by the meter at the downstream reservoir meter continued to be recorded at 15 min (via SCADA)

After time series assembly, the dataset was available for calibrating and validating the VCDM using data from October 2009, capturing the flow trials as shown in Fig. 11.

Fig. 11

Shear stress and measured downstream turbidity dataset from the DI part of trunk main TM-WB. The days on which flow trials 1 to 6 (see Table 2) were conducted are shown as dashed lines with continuous turbidity data available from March 2012. Note y-axis is up to 60 NTU as a valve issue at the monitoring reservoir during trial 3 resulted in a short duration but extreme 50 NTU turbidity spike

The model was fitted to the dataset using the same settings used when fitting the TM-NR dataset. The FIT_SIMULT method was executed three times with the same inputs to assess the repeatability of the PSO fitting process.

Results

Results of fitting the VCDM to DMA flushing events

The results of the five FIT_SIMULT runs for VCDM fitting to the four DMA flushes are presented in Fig. 12 and Table 3.

Fig. 12

Results of fitting the VCDM to four DMA flushing datasets. T_us and T_{hyd_91_obs} are the turbidities observed at the upstream end of modelled pipe reaches, T_{inter_obs} are turbidities observed at intermediate points along the mains and T_ds and T_{flush_wo_obs} are the turbidities at the downstream extent of the modelled pipes. T_{ds_pred_i} (for i ∈{0 … 4}) are the turbidity predictions generated by five PSO runs per dataset (sometimes sufficiently similar such that only T_{ds_pred_4} appears to be plotted)

Table 3 Average and standard deviation results of fitting the VCDM to the PODDS-CI1 dataset five times using PSO. The fitted variables were the β_e and α VCDM parameter values, the maximum layer strength, τ_max, and the initial shear strength, τ_c. Also shown is the relative similarity between prediction and observation time-series using the Nash Sutcliffe (Model) Efficiency Index (NSEI) and the number of iterations the PSO algorithm took to converge. 4* shows data after selection due to data quality considerations

The VCDM mobilisation model fitted well to the first three datasets. The best correlations between predictions and observations were similar for the VCDM and PODDS, based on visual comparison of Figs. 6 and 12. Direct numerical goodness of fit comparison is not presented as it is not considered valid due to differences in implementation of PODDS from within EPANET network modelling and VCDM for single pipe lengths. Figure 12 shows the VCDM is capable of simulating mobilisation functionality across different pipe materials and source waters. The ability to fit the fourth dataset, PODDS-PE2, is hampered by data quality. This is due to elevated turbidity during the first step in this trial that the VCDM model was unable to replicate as part of the complete sequence. For the second and third flow increments VCDM simulations are arguably better than the original PODDS results. The difficulties of fitting this data were also observed in Husband and Boxall (2010), who noted this elevated initial response is a likely consequence of disturbing the system when cleaning the hydrant prior to attaching monitoring equipment. The overall results therefore support a good general fit between VCDM and PODDS simulation and measured response to all pipes investigated. The fitting process was repeatable when the data quality was good, confirmed by the small parameter ranges for three out of the four cases. However, when the quality was suspect, repeating FIT_SIMULT multiple times and visually inspecting the bias of the fits allowed some confidence in parameter estimation. Data quality was assessed subjectively by looking at the distinct nature of both the hydraulic disturbance and the turbidity response relative to background flow and turbidity. Signal to noise approaches with metrics were tried, but a more complex process is required to capture what is currently an expertise based assessment.

Results of fitting the VCDM to TM-NR

FIT_SIMULT parameters for the fitting approach are shown in Table 4, as are the ISE and R² correlation metrics and the reciprocal of the (mean or constant) β_r, expressed in years. There was no difference in any of these values for 3 repeated optimisation runs. Figure 13 provides 6 (from 39) randomly chosen event examples, identified using the automated approach.

Table 4 Details of the best model calibration fits for all DWDS pipe lengths modelled. The first set of parameters for each pipe system is the result of trying to fit to the earlier part of the flushing phase and the second set from trying to fit to the later part

Fig. 13

VCDM fits to trunk main TM-NR (three identically configured PSO runs)

Although the R² for the full times series is only 0.311 it should be noted that the data set is dominated by near zero turbidity data with only small errors over this period dominating the metric. Visual assessment of turbidity simulations during the major shear stress events indicates that the model does represent the key discolouration events. There were however periods with less correlated fit and turbidity events that the model could not account for. There are several possible reasons for this, including: the net turbidity response using T_{ds_obs} rather than calculated from both T_{us_obs} and T_{ds_obs} so material imported from upstream was not accounted for; the flow monitoring not recording short duration events due to fifteen-minute logging, but the response being evident in the turbidity responses. Although the mobilising hydraulic event is too short to be captured at 15 min resolution, material is still mobilised and propagated downstream. Without confirmation of possible short-term hydraulic events, the model is unable to predict this turbidity response. This then has knock on effects into the next event as the amount of accumulated material will be overestimated. The accumulation rate indicates full material accumulation, or maximum discolouration risk, in ~ 1.8 years.

Results of fitting the VCDM to TM-WB

Calibration

FIT_SIMULT parameters from the fitting approach applied to TM-WB are shown in Table 4. Results for repeated optimisations are identical other than slight variation in number of iterations (104, 99, 109). The parameters identified by the three repeat runs are consistent with a reasonable R² = 0.7 correlation with slightly transformed observations.

Prediction

The accuracy/validity of the calibrated model was explored by predicting the turbidity response over the part of the dataset not used for calibration (after trial 6 on 2011-06-22 at 09:12:17 to 2013-10-17 at 13:00:00, approximately 848 days). The values of φ (τ, t) from the final time step of the calibration simulations (see Fig. 14; no observable variation between runs) was used as the wall state boundary condition when simulating from 2011 to 06-22 at 09:12:17 onwards. The relationship between shear strength and relative material quantity at this time is not trivial compared to the PODDS model that is defined by a single value. This supports the VCDM design decision of allowing relative material quantity to vary freely (or at least monotonically increasingly) with shear stress.

Fig. 14

Trunk main TM-WB: the relative material quantity versus shear strength relationship at the end of the VCDM fitting attempt

The correlation between T_{net_est} and T_{net_pred} over this period was not as strong (R² was 0.041 when considering timestamps where T_{net_est} was not null) as for the period used for fitting the model. Figure 15 provides a random selection of event examples. As with TM-NR these figures show visually acceptable simulation results around events, while the correlation metrics are dominated by small errors over the majority of the near zero turbidity data.

Fig. 15

Predicting turbidity in trunk main TM-WB using the model calibrated during the fitted attempt. Red crosses are flow events, identified by an automated method, that are likely to cause material mobilisation

Fitting the model to six distinct flow trials in TM-WB, the FIT_SIMULT method proved to be highly repeatable, both in terms of identified parameter values, and net turbidity profiles with an accumulation rate of 0.9 years. Although reasonable, model fit could have been improved had upstream turbidity observations been available for determining which turbidity increases were due to imported material.

Results summary

Details of the best calibration fits for all six DWDS pipe lengths modelled are shown in Table 4. The model was able to represent DMA mobilisation events in unlined CI pipes well. The fits to mobilisation events in plastic DMA pipes were not as good, even if the maximum shear strength is explicitly limited, but the same was also true for the PODDS model (Husband et al. 2010).

Discussion

An empirical model is preferable to a process derived fully deterministic model given the large number of factors influencing discolouration. The material quantity versus shear strength relationship is considered to be sufficiently complicated for it not to be possible to learn model structure solely from data through empirical machine learning. The most suitable approach is therefore suggested to be a grey-box empirical model consisting of anthropogenic formulations that have been informed by multiple field and laboratory studies.

The VCDM is shown to be able to represent mobilisation events as well as the field-validated PODDS discolouration model. It is therefore proposed that the VCDM succeeds the PODDS discolouration model: it provides the same functionality with fewer parameters and also includes an accumulation mechanism that far more closely resembles observed behaviour. The complexity of the VCDM wall state model is greater than that of the PODDS model, but only as complex as is required to represent the observed characteristics of material mobilisation and accumulation processes.

By understanding factors influencing accumulation, mitigating steps in treatment or chemical dosing can be investigated with the potential to improve future water quality and reduce operating costs. It might be reasonable to expect temperature to impact accumulation rates as a result of effecting network microbial and chemical reaction rates. If model parameters are also shown to be reproducible, the potential ability to transfer between sites could reduce the need for trunk specific studies thereby facilitating widespread application.

While the substantial advantage of the modelling approach verified here is the ability to simulate accumulation with the complexity of the wall bound state suitably described, the processes of accumulation themselves are simply captured in a single pipe specific coefficient. This could be readily advanced as understanding of these processes are better understood, such as including a seasonal temperature dependence, reflecting the understanding that microbial processes are central to material accumulation and that microbial processes are nearly all temperature dependent. Previous work has also shown that accumulation rate is a function of both the bulk water quality in the pipe of interest and the pipe material itself (Husband and Boxall, 2011 and Cook and Boxall, 2011). The dataset for calibration to TM-NR with an upstream turbidity record could be used to indicate if change in background bulk water turbidity over a trunk main pipe length could be a related to accumulation processes. However, dependence of the accumulation rate to background concentrations of turbidity, or any other parameter, was not included in the model form as the evidence for this is currently at best incomplete. Interestingly the bulk water concentration examples discussed here demonstrate that even the understanding of whether, where and when accumulation mechanisms are rate of supply or process limited is currently unknown.

Recent industrial initiatives have stressed the importance of striking a better balance between capital and operational expenditure through consideration of whole life costs. As a result, more water service providers are appraising and developing risk based management strategies to help prolong the life of assets to safeguard water quality. For example, operators may want to understand the impact of a near-term future flow increase, or by how much they could increase the flow without exceeding a turbidity threshold. Using the VCDM this could be posed as a simple optimisation problem. Alternatively, operators could be presented with a real-time management outputs demonstrating the maximum turbidity that will result from any number of instantaneous flow increases.

Data driven methodologies are also being developed that could be used as early warning systems enabling beneficial proactive management strategies to be implemented as an alternative to expensive trunk mains cleaning programs (Meyers et al. 2017, Kazemi et al. 2018). Both data driven methodologies and the VCDM however require large temporal datasets with a range of turbidity events for calibration or facilitate machine learning training. The VCDM is based on conceptual understanding, so if further calibration is shown to offer grouping of VCDM parameter values, and hence transferability between locations, it would potentially be of generalised value. Both approaches have the potential for greater operational application and there is a trade-off to be explored based on amount of data and under/overfitting.

The VCDM could in future be refined using knowledge gained from calibrating model instances for a variety of different systems and also by using machine learning techniques to build accumulation sub-models. The resulting improvements in predictive accuracy could drive the uptake of asset management strategies where shear stresses are periodically, pro-actively increased so that planned or unplanned flow increases do not cause unacceptable turbidity responses. By tracking discolouration potential over an extended period (weeks, months or years), the VCDM can equip water providers with a tool for assessing how different interventions and incidents could impact bulk water turbidity, alert operators when a flow increase is predicted to cause the bulk water turbidity to exceed a threshold, review network changes by analysis of accumulation rates, develop and optimise flow conditioning programmes for non-invasive cleaning and determine associated whole life costs.

Following the quotation, or slight misquotation of George Box, we fully accept that the model we propose is in one sense wrong, in that it does not fully capture or describe all the processes and interactions occurring. However, the majority of the validation presented here shows the ability to simulate the observed turbidity data to an accuracy of less than 1 NTU. This is the turbidity standard applied at the exit of water treatment works, is significantly less than the 4 NTU limit at consumers taps and even lower than the generally accepted limit of human observation which is, subject to lighting conditions and size of sample, generally into the teens. Hence we conclude that the model is useful.

Conclusions

The paper presents calibration and validation of a semi-empirical model that tracks the continual processes of accumulation and mobilisation of material from pipe walls within drinking water distribution systems. Functionality to enable automated calibration was developed and implemented making use of particle swam optimisation. This was found to provide efficient and repeatable solutions from rigorous searching of the problem space. Automated approaches were also developed enabling identification of periods of interest from long and complex turbidity time series data.

The model formulation was shown to maintain the mobilisation functionality of previous validated shear-stress-dependant modelling tool, but requiring only two empirical parameters. Evidence of this was presented for four classic cases across both cast iron and plastic pipe materials and hence covering different material behaviour within these. Calibration to long time series flow and turbidity data was presented for two trunk main systems. The model is shown to make usefully accurate simulations for flow mediated events and with sufficient data, validation of the predictive capability was demonstrated. This supports the observation and model functionality that material with all shear strengths accumulates continuously at a rate invariant to hydraulic forces.

Overall the paper demonstrates the capability of the new formulation combining a sub-model tracking the temporal shear strength profile of accumulated pipe wall material and a hydraulically defined material mobilisation model. With the VCDM, water service providers have for the first time a tool to assess both effective real-time and long-term hydraulic discolouration responses, quantify network changes and plan, with costing implications, maintenance schedules to safeguard drinking water quality.