Report: Review of the evidence on indicators, metrics and monitoring systems

The report examines 20 years of monitoring initiatives in sustainable agriculture. It provides insights and tools to help stakeholders prioritise investments and manage competing development goals. It is designed to inform future DFID research investments and assesses both biophysical and socioeconomic related monitoring efforts.  With the aim of generating insights to improve monitoring systems, the report focuses upon key questions facing stakeholders today:

• How to evaluate alternative research and development strategies in terms of their potential impact on productivity, environmental services and welfare goals, including trade-offs among these goals?
• How to cost-effectively measure and monitor actual effectiveness of interventions and general progress towards achieving sustainable development objectives?

Findings and Conclusions

A total of 103 monitoring initiatives were identified and analyzed. The report identifies opportunities that exist to increase return and it ends with a series of clear steps and opportunities on how to transform these recommendations into reality to help make more informed decisions on agricultural intensification that reduces hunger and poverty sustainably for generations to come.

Some weaknesses of current initiatives include:

• There is a lack of a clear conceptual framework that could demonstrate an understanding of the system under study.
• There are problems in most initiatives in defining the target inference space (geography, population) and how that is being sampled.
• Very few initiatives have come to terms with how to integrate biophysical and socio-economic indicators and sampling frames.
• There is a lack of welldefined sample units or strata and a lack of consistent measurement protocols.
• Use of statistically sound study designs to allow attribution of outcomes to interventions is still rare.
• There is a pervasive lack of consideration of uncertainties both conceptually and in communicating results and few initiatives have tackled trade-offs among objectives.
• Evaluation of monitoring initiatives themselves is lacking and there is virtually no information on cost-effectiveness of the measurements or the impact of the initiatives.
• Few initiatives have been sustained over the long term pointing to inadequate consideration of institutional sustainability.

The primary lesson learned is that despite investment of massive amounts of resources in monitoring initiatives, there is a surprising lack of evidence for their actual impact on decision-making or management, except in cases where initiatives have been explicitly designed to inform specific decisions (e.g. regulation). This may be a principal reason why few monitoring initiatives are actually sustained.

A set of recommendations for good practice in monitoring initiatives is given and opportunities for new thinking in monitoring design are identified, including a decision analytic conceptual framework. Some of these were:

• A clear conceptual framework to demonstrate an understanding of the system under study. In particular theories of change on how the monitoring results would affect behaviours and explicit linkage to specific decisions are weak or lacking.
• Consistent and well-documented measurement protocols, so that there is opportunity for aggregation and meta-analysis of results towards the development of generalizable knowledge and provision of a reliable picture of state and trends.
• Build scale hierarchy explicitly into the sampling design and statistical analysis methods, which is particularly critical for decision-making on sustainable agricultural intensification. Tools for doing this, for example through multilevel sampling, and use of mixed effects statistical models, have recently become more easily accessible.
• Determined efforts to integrate biophysical and socio-economic indicators both conceptually and in sampling frames. A particular challenge is how to link sampling units used in biophysical monitoring initiatives (e.g. fixed area sampling or watershed delineations) with units commonly used in socio-economic monitoring (e.g. households, villages).
• Designs that allow attribution of impacts of interventions. Use statistically sound study designs where possible. Disaggregate indicators across different levels of important conditioning variables (e.g. by gender, income group). Monitor variables along the impact pathway to accumulate evidence of intervention impacts.
• Make data and information generated by research and government institutions accessible and reduce costs associated with access.
• Put in place active mechanisms for dissemination of results to target audiences, beyond web-based dissemination.
• Collect relevant data to be able evaluate the impact and cost-effectiveness of monitoring initiatives, to help make a better case for sustaining initiatives.

Citation as follows:

Shepherd, K.D.; Farrow, A.; Ringler, C.; Gassner, A.; Jarvis, D. Review of the Evidence on Indicators, Metrics and Monitoring Systems. World Agroforestry Centre (ICRAF), Nairobi, Kenya (2013) 94 pp.

For the full Review of the Evidence on Indicators, Metrics and Monitoring Systems report click here.

You can also have a look at this blog-post by CGIAR here.