Addressing uncertainty in WCPFC stock assessments: Review and recommendations from WCPFC Project 113

Model weighting is a central challenge in stock assessments, because the retention or rejection of models, and relative weights given to models and their respective uncertainties can markedly affect quantities measuring risk of available management options. In the Western and Central Pacific Fisheries Commission (WCPFC), there are currently no explicit terms of reference that guide the development and subsequent weighting of model ensembles. For this reason, a range of approaches for developing stock assessment models and model ensembles have been employed, ranging from single base-case stock assessments with relatively few key sensitivities to grid-based ensembles with a large number of models; other assessments employed intermediate approaches that considered a limited number of models.

This research for WCPFC project 113 aimed to provide both general and specific review components to develop recommendations for the presentation of stock assessment and management advice uncertainty by the WCPFC scientific committee. The terms of reference for the general review were to:

1. Review and summarise the different approaches used for characterising uncertainty in WCPFC stock assessments for tuna, billfish and sharks over the last five years.
2. Describe how uncertainty was communicated in the context of management risks and its influence on decision-making processes used by the WCPFC.
3. Comment on the suitability of the recent approaches to characterising uncertainty for the management systems, including the harvest strategy approach.

The specific review aimed to:

1. Critically review the ensemble approach that was applied for the assessment of southwest Pacific Ocean swordfish assessment in 2021 (SC17-SA-WP-04, Ducharme-Barth et al. 2021) to capture both “structural” and "estimation’’ uncertainty.
2. Conduct a similar review of the approaches used in the analysis pertaining to the stock assessment of southwest Pacific Ocean blue shark (SC18-SA-WP-03, Neubauer et al. 2022).
   
3. Based on these reviews, provide recommendations for model ensemble construction, model retention, and weighting of models included within ensembles in the context of the WCPFC tuna, billfish, and shark assessments.

The expected outcomes of the project were to provide:

1. a basis for stock assessment teams to consider and apply alternative approaches for characterising stock assessment uncertainty (including model selection and weighting) across the WCPFC tuna, billfish, and shark assessments;
   
2. guidance for the Scientific Committee (SC) about the approaches
   for capturing assessment uncertainty in the provision of management advice; and
3. improved understanding for managers and stakeholders of the implications of alternative approaches to characterising uncertainty for their perception of risk.

Based on these term of reference, the two most recent assessments for all stocks considered by the WCPFC SC were reviewed using a structured approach. The review focused on the stock status and management advice as provided by the SC. It considered how uncertainty was addressed through the use of ensembles and sensitivities, and whether this uncertainty was used in management advice. In addition to the review, the present study included discussions with working group members of the Pacific Community (SPC) and International Scientific Committee for Tuna and Tuna-like Species in the North Pacific Ocean (ISC) to gain a thorough understanding of the rationale for the approaches used for addressing uncertainty in stock assessments.

The findings from this study revealed clear and long-standing differences in approaches between the ISC and SPC working groups for addressing uncertainty in stock assessments; however, there was a recent rapprochement of approaches. Although the ISC working groups traditionally preferred to present a single model with different levels of uncertainty for management advice, their recent advice included a more explicit consideration of alternative models and estimation uncertainty. In contrast, some SPC assessments previously used a considerable number of models in “structural uncertainty grids”; these assessments were without explicit consideration for a single "best’’ model, often considering all models in the grid equally plausible. Nevertheless, more recent assessments by SPC have attempted to constrain these model grids to sets of plausible models, and have explored metrics to weight models.

There is currently no best practice identified to weight models, or to address uncertainty in stock assessments more broadly, but there may be a valuable "`middle ground’’ for both aspects of stock assessments. This approach would explicitly acknowledge and explore uncertainty, with standardised reporting to allow consistent management advice of the uncertainties considered for each assessment.

To illustrate some of the observations and recommendations from the current analysis, a set of simulations was set up. The simulations were designed to highlight differences in approaches to the characterisation of uncertainty, and were not intended to be a a realistic representation of typical stock assessments.

Based on the current review and simulations, we developed a set of recommendations for consideration by the SC19 relating to the use of model ensembles for management advice, and for the communication of assessment uncertainty.

Model ensembles and weighting

1. Develop joint priors and explicit rationales for grid axes and their values.
2. Either draw from, or weight axes over parameters according to the joint prior.
3. Consider observation error, structural, parameter, and estimation uncertainty in management advice.
4. Where possible, express priors for model outcome space to avoid post-hoc selection/weighting.
5. Where post-hoc weighting is necessary (unexpected outcomes), this weighting should be proposed by analysts.
6. Clarity about uncertainties addressed by the grids address, including clear and consistent terminology around uncertainty.

Communicating uncertainty and risk

1. Develop a template for reporting management advice and uncertainties; ideally this template is a standardised table format to help managers and stakeholders locate key quantities easily.
2. Agree on terminology and the set of required measures (ideally probabilities relative to reference points).
3. Clear communication about quality of information determining stock status and management advice:
   • Qualification and quantification of uncertainties.
     1. Data quality.
     2. Model/population: structural uncertainty. (Note the use of “structural” here refers to models with different likelihoods, rather than different parameter values.)
     3. Key parameters (parameter and estimation uncertainty).
     • Key uncertainties and potential impacts.
4. With respect to item 3, develop a set of research recommendations to address key uncertainties.
5. A review of timelines and capacity for tuna stock assessments may be necessary to allow sufficient time and capacity to adequately address uncertainty. Sufficient time is also needed to enable the provision of management advice that is consistent with the application of the precautionary approach as outlined in the WCPFC convention text.

Further development and future research

In addition, we suggest that the SC19 consider recommending:

• the provision of a project to develop a standardised reporting template for the reporting of uncertainty and risk that incorporates recommendations made in the present review; and
• the further development of methodology and idealised simulations to develop principled model ensemble approaches, in particular to consider the ability of alternative model diagnostics to identify model plausibility and weights.