Multi-method evaluation to understand food system interventions

Summary

This article uses a mixed methods approach to evaluate supply-side efforts to raise consumption of nutritious foods. A case study in Kenya is used to open up a broader discussion on the impact and evaluation methods of complex food system interventions. The authors note a rising interest and use of ‘food system’ approaches but do not specifically define said approaches.  

The authors focus on the food supply chain which they define as “the processes involved in getting food from the farm to the consumer and encompasses food production, processing, distribution, and retail.” They also discuss the role of the food environment, or as Turner et al. (2018) define it, “the interface that mediates people’s food acquisition and consumption within the wider food system,” on food consumption through food availability, affordability, and desirability.

The authors claim that previous ‘traditional’ research on impact of food supply interventions for improved nutrition such as randomised trials have been marred with methodological and complexity issues; with direct causality difficult to evidence. They call for an expansion of methods on the subject which build from food systems approaches and explore new methodological approaches from outside traditional nutrition studies. 

The authors attempt this through the analysis of a food supply intervention in Kenya. They evaluate the impact of the intervention and the methodological approach that was taken. The specific intervention was the Marketplace for Nutritious Foods (MNF) project, implemented by the Global Alliance for Improved Nutrition (GAIN), which was designed to increase consumption of nutritious foods through the support of food producing firms. These firms were classified as small and medium-sized enterprises (SMEs) which produce and sell foods considered nutritious. The project aimed to expand their production and extend the reach of their products within low-income populations.

The MNF has two components: the Community of Practice (CoP) and the Innovation Accelerator (IA). The CoP sought to facilitate a learning environment for stakeholders related to business and nutrition and a networking environment to foster information sharing and mentorship. This was done through meetings, trainings, newsletters, and communications on online platforms. The IA accepted competitive proposals from prospective SMEs. A committee then selected the most promising proposals to provide technical and financial support. It aimed to improve management practices, production capacities and marketing strategies and provided grants for equipment and infrastructure. The MNF also sought to evaluate the impact of the overall project on nutrition and consumer food access in the study region.

Due to constraints (short two year timeframe, small number of firms involved and non-random selection of firms) and the complexity (the intervention was filtered through multiple steps before changes were enacted so casualty of changes is difficult to discern) of the project, a simple evaluation method was deemed insufficient. The authors drew from program theory and designed an evaluation method which breaks the program’s impact into several streams which contain smaller components or units of evaluation. Different project components were assessed using different methods which were deemed most appropriate by the authors to answer guiding research questions. Figure 1 demonstrates this method.

Figure 1: Intervention project evaluation method using program theory to assess components of the program non-sequentially. 

To answer the three primary research questions, a mix of quantitative and qualitative methods were used. A survey and an economic game were used to assess if the CoP increased networking, trust, and cooperation. The survey alone did not fully address the level of network expansion of participants due to a lack of control for overlapping and pre-existing networks. The authors do report, however, that many respondents felt like they made new connections and expanded their networks through the CoP. During the economic game, participants received an initial ‘endowment’ and were broken into two groups - senders and receivers. A sender could make a ‘joint investment’ to a recipient who would receive triple the amount sent. The receiver then was able to choose how much to return to the sender. Joint investments are collectively profitable but larger investments require more trust. The game thus assessed trust and cooperation. The main finding was that established personal relationships enhanced trust of participants. Paired with the survey, the results suggest that the CoP could contribute to better cooperation between firms. The authors do not, however, address whether this cooperation and trust would be maintained in the absence or collapse of the CoP or the relative longevity of an intervention like the CoP.

The authors used a statistical analysis to address if technical and financial assistance from the IA improved firm performance and the launch of new food products. They found it was successful at supporting SMEs in launching new products (30% more likely than controls) and employing more people (7-15 more workers than controls). The authors found no significant evidence to suggest the IA had an impact on sales, profits or assets. Recipients of IA assistance were also interviewed and generally found the program to be positive; indicating an improved understanding of market demand, business planning and other management practices. Participants also reported they believed they were able to expand and increase sales and profits despite statistical evidence not supporting this. To assess if availability of nutritious food leads to increases in consumption, the researchers experimentally simulated a “best-case” scenario for the increased availability of a nutritious food item. The experiment was “real” but the researchers artificially created an environment in which a specific food was "more available." Instead of waiting for upstream interventions to increase availability of a food item in retail stores, they gave some SMEs in the study a food item to a level they deemed as sufficiently "available” and left other stores without the item as controls. The nutritious food item was a peanut butter sachet (no added sugar or oil, and reduced risk of aflatoxin contamination). Unsurprisingly, nearly all of the retailers who were given the product sold the product and those who weren’t did not. More interestingly, both consumers in treatment segments and those in areas of control reported increased consumption of the product (16.3% versus 8.3% respectively). The result suggests wider availability of a nutritious food can increase the consumption of that food. The authors do note that overall peanut butter consumption did not increase, suggesting a replacement of other peanut butters with the experimental peanut butter sachet. 

The researchers used a household survey and a hypothetical price experiment for food items with participants to assess if greater availability of nutritious food increased willingness to pay for it. The authors found that participants were willing to pay more for labelled products compared to unlabeled products and especially preferred products labelled with positive health and safety information. Different demographic groups also are more willing to pay for products that are labelled with health and safety information specific to them such as women and households with children. The authors note these hypothetical choice experiments are limited by their lack of real world setting and thus aren't generalisable. 

The authors then assess the overall effectiveness of MNF intervention. They found that the CoP and IA generally support the launch and production of new nutritious foods and increased employment but did not find a direct link to increased sales, profits or assets. They also found mixed results for the impact of the project on consumption. Overall consumption of the experimental food type (peanut butter) did not increase, although the results suggest the more nutritious option may have replaced less nutritious alternatives. It also did not affect household dietary diversity. In other words, availability did not increase desirability in this study. Generally, the authors seemed to look at the economic and nutrition impact of the MNF intervention but did not give robust attention to how this intervention affects, and is affected by, the whole wider food system in which it is a part. For example, the authors did not provide a robust analysis of how the interventions would impact farmers, harvesters, energy and water needs and the environment.

The authors conclude that overall, due to time and scale constraints of the MNF, similar projects  are unlikely to generate statistically significant market- and consumer-level effects. The authors provide five suggestions for larger-scale and more systematic approaches. 

1. More significant financing interventions (impact investment) would support access to finance for more firms at various points of the supply chain, increasing the impact beyond what is possible through a grant-based approach like MNF.
2. “Whole-of-supply-chain” approaches (or those that work with actors along a supply chain and not just at one part of the supply-chain) may have greater impact, especially where there are issues in the supply chain, such as difficulties sourcing safe food inputs.
3. Availability should be coupled with demand-side interventions (such as targeted marketing campaigns) to influence desirability and willingness to pay for more nutritious foods.
4. Marketing campaigns alone are insufficient to effectively improve the diets of the lowest-income consumers; monetary support is required to promote purchasing of nutritious foods.
5. Food system policies must reduce business costs. Examples include strengthened contract enforcement; increase access to credit, such as collateral registries; address supply chain bottlenecks that constrain growth or increase prices; and provide clearer information to consumers on which foods are and are not nutritious, such as front of pack labelling.

Abstract

A ‘food systems’ approach to improve diet quality by intervening within areas such as food supply chains is gaining prominence. However, evidence of such interventions’ impact, and understanding of appropriate methods to evaluate them, is lacking. We present an impact evaluation of an intervention that aimed to increase consumption of nutritious foods by supporting food-producing firms in Kenya. In doing so, we demonstrate how multiple methods, including those from other disciplines, can be used to evaluate a complex food systems intervention. Four methods focused on food-producing firms and their management, including a survey of intervention participants (n=83 individuals), a ‘lab-in-the-field’ experiment (n=83 individuals), baseline/endline data on firm performance (n=71 firms), and semi-structured interviews (n=19 firms). Three methods focused on consumers in neighborhoods targeted by a supported firm: a randomized field experiment tested effects of making a supported product exhaustively available on consumers’ purchases and consumption (n=1,295 consumers); three discrete choice experiments (n=1,295 consumers) tested factors influencing consumers’ willingness to pay for foods with relevant characteristics. Among firms, we saw suggestive evidence of increased networking and business relationships, while lab-in-the-field experiments indicated the intervention might foster cooperation among participants. Qualitative interviews suggested that the intervention enabled firms to increase production, improve management, increase revenues, and lower costs. Baseline/endline data confirmed a positive effect only on the launch of new products and hiring workers. In the field experiment, consumption of the supported product increased in areas where it was made available relative to a control group, but this did not increase overall consumption of the food type or dietary diversity. Results showed positive signs of the intervention improving firm-level outcomes but limited impact on consumers’ diet quality. The evaluation also demonstrates how diverse methods can be used to evaluate complex interventions.

Reference

Maredia, M.K., et al. 2024. Using novel multi-method evaluation approaches to understand complex food system interventions: Insights from a supply chain intervention intended to improve nutrition. Current Developments in Nutrition 103776.

Read more here. See also the TABLE explainer What are food systems? and our report on Investment, Power and Protein in sub-Saharan Africa