The CEDARS Center / SHOUT Group Blog

The CEDARS Center / SHOUT Group Blog
Return to CEDARS' Home Page.
Join the SHOUT Group by registering on CEDARS to be allowed to upload projects and experiences.
Contribute to this blog.

Wednesday, November 25, 2015

Training Opportunity

CEDARS tends to focus on health-in-development and related topics, and we have focused a lot of attention on design, management and evaluation processes to enhance sustainability, making use of quantitative data as much as possible, while being firmly anchored in implementation.

Here's however a major training opportunity for people who are interested in the hard science, the quantitative underpinnings of some of the major issues in sustainability. No better place to do it than the Santa Fe Institute. The focus is on urban sustainability, but it's probably a good opportunity to learn about methodologies which we need to pay more attention to.

Here's to the young innovators ready to learn new things. And if you're not that young, you just qualified by virtue of thirsting to learn.

Check out the course information here.


Friday, January 30, 2015

General Relativity Comes to Global Health—thinking about sustain-scale in health systems interventions

ICF International Center for Design and Research in Sustainability | Sustainable Health & Human Development (CEDARS), January 2015

A metaphor for sustain-scale?
I want to offer a metaphor to question some of  the ways that we, as the global health  community, have been talking about and dealing  with two concepts: sustainability and scale.

I need to start with a little bit of recent history…

Tuesday, November 4, 2014

Resources for adaptive management practices and cost-effectiveness in development

We recently added two new documents to our CEDARS Center resource repository to help development professionals think about adaptive management practices and program cost-effectiveness during implementation, planning, or evaluation.

Have a read through below and click through the links. As always, should you have comments or questions (or additional resources we can share with the sustainable health and human development community), do not hesitate to reach out to us.

Navigating Complexity: Adaptive Management at the Northern Karamoja Growth, Health, and Governance Program [document available here]

This paper, by Engineers Without Borders Canada, under contract with Mercy Corps (MC), is a case-study regarding adaptive management practices within Mercy Corps’ (USAID-funded) project, Growth, Health & Governance Program (GHG). The paper covers building the culture necessary for learning and adaptation, discusses some tools and processes that support adaptation, and some implications for funders and practitioners. Throughout the document, culture is emphasized as the most important factor to be successful in adaptive management and provides strategies and attitudes deemed necessary in achieving this culture. The tools and processes are presented with the purpose of reinforcing the described culture. One of the tools, the Results Chain, is an interesting way of conceptualizing the path to reaching the goals of the project and is similar to a results framework.

A blog post, with a summary is available here:

Cost-Effectiveness Measurement in Development: Accounting for Local Costs and Noisy Impacts [document available here]

This policy research working paper from the World Bank Group, Africa Region, can help individuals think about cost-effectiveness within their programs from implementation, planning, or evaluation perspectives. As evidence from rigorous impact evaluations grows in development, there have been more calls to complement impact evaluation analysis with cost analysis, so that policy makers can make investment decisions based on costs as well as impacts. This paper discusses important considerations for implementing cost-effectiveness analysis in the policy making process. The analysis is applied in the context of education interventions, although the findings generalize to other areas. First, the paper demonstrates a systematic method for characterizing the sensitivity of impact estimates. Second, the concept of context-specificity is applied to cost measurement: program costs vary greatly across contexts -- both within and across countries -- and with program complexity. The paper shows how adapting a single cost ingredient across settings dramatically shifts cost-effectiveness measures. Third, the paper provides evidence that interventions with fewer beneficiaries tend to have higher per-beneficiary costs, resulting in potential cost overestimates when extrapolating to large-scale applications. At the same time, recall bias may result in cost underestimates. The paper also discusses other challenges in measuring and extrapolating cost-effectiveness measures. For cost-effectiveness analysis to be useful, policy makers will require detailed, comparable, and timely cost reporting, as well as significant effort to ensure costs are relevant to the local environment.

You can find additional information on this paper here:

Tuesday, September 2, 2014

Learning from a post-project evaluation study, applying systems thinking and addressing complexity in community health

By Michelle Kouletio

It doesn't happen enough, but every once in a while a seed of a sustainable health intervention is planted in the ground.  In this case, the seed was planted at the doorstep of a mayor's office in northwestern Bangladesh, amid narrow, busy roads with open sewers of common of bustling secondary cities. Among the middle class families live the extreme poor.  Along with several other challenges, the poor are failed by the government health system whose facilities are overwhelmed by patient volume and whose outreach workers do not serve.   While national policy assigned elected municipal leaders with responsibility for ensuring coverage of equitable health services, these leaders were provided little guidance nor resources.

With support from the USAID Child Survival and Health Grants Program, Concern Worldwide worked in this and other municipalities in Bangladesh to empower municipal leaders to develop a replicable model for social mobilization in complex urban environments. What made this project unique from so many well intended and ambitious community health projects was the embedment of a systematic sustainability planning and monitoring system that established a shared vision and measurement framework across the Mayor’s office, elected representatives, service providers, social leaders and health volunteers along with the project team.

As the technical advisor for Concern Worldwide, I had the privilege of backstopping this initiative.  It took quite of bit of extra effort, particularly in developing practical capacity measuring tools from scratch and maintaining regular reviews at the neighborhood and municipality level. Sustainability planning also required tackling structural barriers and inter-ministerial relations which could have otherwise been ignored in a conventional project. However, the benefits of this deeper analysis and shared accountability approach resulted in real improvements in equitable health outcomes and an enduring political mobilization approach that allowed the population to continue reaping benefits years after the project closed.

Two recently published articles on this work further validate the importance of the hard work of the Concern Worldwide staff in Bangladesh and their contribution to the developing body of evidence on adaptive health systems and sustainability planning:

Post-script from Eric Sarriot:

These two papers coincide in their release to form a useful series on the Concern CSHGP Bangladesh experience. The first one is part of a larger and important Supplement of Health Research Policy and Systems on Systems Thinking in Health, coordinated by Taghreed Adam of WHO’s Alliance for Health Policy and Systems Research.

Thursday, August 21, 2014

Putting vulnerability first – the need to revise targeting and delivering aid to promote sustainability and resilience

By Debra Prosnitz, MPH

The Humanitarian Policy Research Group (HPG) recently released a policy brief on resilience: Political flag or conceptual umbrella? Why progress on resilience must be freed from the constraints of technical arguments (1). Reading this brief, I began to reflect on the relationship between resilience and sustainability. In the context of development, resilience should address ability to cope and recover from crisis, and the sustainability should address the process of strengthening social capital to sustain progress in health, social progress, etc. (2). If sustainability is a process, perhaps resilience is one of the outcomes. I decided that the importance of this topic deserved a CEDARS blog update.

In this short (4-page) brief, Simon Levine succinctly captures two approaches to addressing resilience, brings to light the stalemate and consequent inaction in which they are stuck, and suggests a new way forward in theory through action (3).

Levine discusses two “broad arguments” for addressing resilience, both of which, he argues, distract us from the underlying need to identify and understand vulnerabilities of individuals and communities, and find ways to address these vulnerabilities: The political argument convinces us that something needs to be done - “since the shocks and stresses that cause crises cannot be prevented, the task is to ensure that people are better able to cope when things do go wrong,” and the technical argument calls for a refined definition of resilience and new approaches for addressing it because what has previously been implemented has not adequately addressed the complexity and challenges of resilience, such as climate change. While we may not yet know the best way to define resilience or the best strategies and approaches to address it, we must not let this stop or delay efforts to do so.  We know that vulnerabilities exist and should be addressed, and should move forward by identifying and understanding vulnerabilities of individuals and communities, and findings ways to address these vulnerabilities.

Resilience then—even if imperfectly conceptualized--can already be enhanced by ensuring that “vulnerabilities are the center of development policy and investment,” and that marginalization be addressed and minimized through development; resilience should not become another sector of development. Thus, Levine suggests that the definition of and theoretical framework for resilience can be developed and refined through action (rather than delaying action pending conceptual clarity).

While Levine addresses the complexity of development and the “structural, institutional and bureaucratic obstacles” to making change to how development aid is targeted and delivered, I wish he had made a clearer link between resilience and sustainability. I’m envisioning resilience as a pillar of sustainability, because while we can’t predict which shock will occur, the occurrence of shocks and changes following even the best of our interventions is almost a certainty. Further, neither resilience nor sustainability can be achieved by external intervention and leadership alone. Community involvement should be a prerequisite for approaching both, with communities and individuals leading efforts to identify vulnerabilities and define resilience.

Levine’s brief is a glimpse into larger bodies of work he has published on this subject (4), which can also be found on the HPG website. Whatever your stance on resilience as a concept, this paper is an important reminder of the old adage not to let the perfect become the enemy of the good; we should not be complacent in our thinking about models of aid delivery, and should be actively thinking about and advocating for changes in the way in which aid is both targeted and delivered. Vulnerability should be the primary criteria, with aid targeted toward the most vulnerable first and delivered in a way that bolsters resilience.  

(1) S. Levine, Political flag or conceptual umbrella? Why progress on resilience must be freed from the constraints of technical arguments. Policy Brief 60. Humanitarian Policy Research Group (London: ODI, 2014).
(2) Sarriot, et al. Taking the Long View: A Practical Guide to Sustainable Planning and Measurement in Community Oriented Health Programming. Macro International, Inc. (Calverton, MD, 2008) defined sustainability as “a process that advances conditions that enable individuals, communities, and local organizations to improve their functionality, develop mutual relationship of support and accountability, and decrease dependent on insecure resources…(and) enables local stakeholders to play their respective roles effectively, thus maintaining gains in health and development…”
(3) As our fearless CEDARS leader Eric Sarriot summed it up “praxis can improve without a perfect epistemology.” 
(4) S. Levine, Assessing resilience: why quantification misses the point. Humanitarian Policy Group Working Paper. (London: ODI, July 2014); and 
S. Levine and I. Mosel, Supporting Resilience in Difficult Places: A Critical Look at Applying the ‘Resilience’ Concept in Countries Where Crises Are the Norm, HPG Commissioned Paper for BMZ (London: ODI, 2014).

Tuesday, July 15, 2014

Concise but Important Text on Development Evaluation by Fred Carden

I came across a very interesting and concise article by Fred Carden from RTI in the December 2013 American Journal of Evaluation, which makes plain and simple some important points—whether you’re working from the perspective of sustainability, systems strengthening, country ownership, or learning and adaptation.

Carden’s introductory quote captures his key point about evaluation succinctly: “it’s not about your project, it’s about my country.”

While I would prefer critiquing the emphasis for “international development aid evaluation” instead of Carden’s “development evaluation” (which I think is always relevant—aid or no aid*), Carden makes three simple but important points:
(1) Evaluation will endure but “[international] development [aid] evaluation” is not a permanent field of practice.
(2) Evaluation needs to look at systems, not projects; and
(3) Evaluation requires local expertise.

I think our work on the Sustainability Framework has certainly taken this look at systems before projects to heart, even if it has been a challenge. We have used local expertise, but whether we have given sufficient leadership to local expertise is more questionable. We’ve recently had another experience with this type of local system perspective:

Ilona recently assisted a Gates Grantee in laying the foundation for an evaluation approach looking at the local and national systems rather than just the project, in this case by developing a theory of change. A theory of change (TOC) is a type of logic model that articulates an expected outcomes pathway, the causal relationships, and the underlying assumptions that relate to the broader social, cultural, political, economic, or institutional environment behind the process of change. TOCs are under-utilized tools that can be very helpful in mapping out, in Carden’s words, the “constellation of activities [and we would argue also, of actors] that create change and betterment” in a system. If conceived with that perspective in mind, a TOC encourages a broader view of change beyond the immediate project that encompasses (and is grounded in) the realities of the context and therefore allows key elements outside the project boundaries to be explored and included in the evaluation of a change process. By clarifying how particular activities are expected to produce particular outputs and outcomes (and the relevant assumptions), the TOC helps in framing the evaluation questions to provide meaningful insights or evidence of program success and identify possible counterfactual explanations, as relevant.

Such approaches and the logic behind them are going to continue to grow in relevance. Carden articulates this logic very concisely and clearly—we recommend you take a look at his short (less than 3 pages!) article.


[*] I would suggest just one language adjustment to the first point and state that “international development evaluation” is not a permanent field of practice. Development—as in social and human development-- needs to continue here, there, everywhere and at all times; and all stakeholders from government to civil society need to rely on good evaluation to learn and adapt.

Saturday, June 21, 2014

And finally.....Country Ownership and its Measurement - Part 4 of 4

Preamble: this is the last of a 4 piece series. 
Part 1 dealt with why country ownership matters to us now. (last footnote of this entry not withstanding)
Part 2 dealt with the understanding of ownership from large development aid actors, what our role is in advancing and measuring ownership.
Part 3 discussed dimensions and metrics of ownership, then addressed the methodological choices we must make as happening on a rugged, and even dancing landscape.
This is the 4th and final entry, where I discuss an fundamental choice we must make in our measurement approach. You probably should read at least Part 3 before proceeding -- the graph below is explained in Part 3.

Practically, what does all this this mean?

This discussion may have gotten a little ethereal for some, so let me try to bring it back to the practical and for this, we need to get back to the “why” of ownership assessment and consider our options. 

Imagine then that your options for measurement are between B5 (limited subject engagement) and B7 (limited objectivity), what option should you take? 

Let’s consider the options, starting as far to the left of the spectrum as we can:

We have far more experience measuring capacity than ownership. Given that capacity is conceptually part of ownership, and that ownership is even more of an abstract concept, I’ll stick to a capacity measurement example.

The more informative measures require a high level of locally relevant detail, which is very hard to obtain from the outside. Consider a simple capacity indicator, for example in human resources for health management: the percentage of a specific type of personnel actually available over a given year to perform a specific function. This seems like a very objective measure, somewhere toward B1. But experience shows that to make the indicator most informative (and useful for decision-making), you need to be guided through the complexities of public administration procedures and rules of the specific country. Coming to the proper definition of the type of personnel, what being “in the plan” means, and where the responsibility for ensuring that key positions are filled actually lies, is challenging from an outsider’s perspective. Without having a “guide” into the local health system under investigation, proper and meaningful measurement will be very difficult. Of course it gets more and more difficult once you start asking about things such as shared accountability, institutional ownership and political will.

So, staying on the left side of the spectrum, first you will struggle tremendously finding valid measures—it might be possible for a research exercise, but will certainly be challenging for a time-bound monitoring activity. (I am speaking of very practical constraints, for example getting the right staff in the right office of the MOH administration to help you figure out why this register (rather than the one anticipated) is the right one to get your denominator from.) Even assuming that you do get at measures, which can be considered valid, what happens if you were so far on the left end of the spectrum (B1 or B3) that local stakeholders are unsure of you, what you have measured, and what this even means? What is going to be the value given to your measure? Even for capacity measures, B1 or B3 are going to be uncomfortable spots to be in, unless you have the power to impose an audit and make your own rules.

If we go back to ownership, and from the perspective of why you wanted to measure it, you certainly won’t be sending actionable signals to those constituencies, even if you feel good about your measure. Like a tree falling in the proverbial forest, you might provide a valid indicator, but if no one is here to believe its signal, was it worth it?

You consequently are forced to move toward more participatory engagement of stakeholders to first determine what measures are meaningful and then define them operationally. You have to push through and over B6, and this takes you to B7.

You now have measures for which there is “buy-in” and cultural translation from stakeholders. Presumably, to have gotten such a buy-in you have developed with them a purpose, an action-orientation for your measures. Nobody claims that getting there is necessarily an easy road, but you are essentially working and measuring from some inside-the-system perspective. Having built rapport and a clarity of purpose along with some trust, your external “expert” voice will both carry more weight, as well as be kept in check by actors in the process you are measuring.

Your data might come to inform the stakeholders, but of course now your biggest concern is the external validity of your measure. If you present your findings you will dread questions from researchers with those letters after their name. The question is, having moved from B5 to B7 and gained buy-in and internal validity of your capacity or ownership questions, was it worth it if the external validity of your measure is now challenged?

My best answer at this point is, yes. And here’s why:

1-     The sole reason why you wanted to measure ownership in the first place was to engineer change with these same stakeholders. The first option (B5), leaving these stakeholders to wonder what it is that you measured does not help your goal. If measurement is here to guide change management, what is the value of a better measurement which means little to the change managers?
2-     Your (B7) measurement may have—certainly has—flaws but if it serves to inform and guide an authentic process of planned change, you have a foundation to build upon. Managers deal with uncertainty every day anyway. You will have lost some of the precision on the details, but probably gained validity on the big picture. Actually, actors of the local system will have an incentive to help you improve your measures over time—measures inform change, but change also informs (better) measures. Your measurement expertise will now be able to support a management process, rather than chase “data use”.

By starting from B7 you can influence a change in the landscape, and the possibility of moving toward better, more reliable measures as you promote more ownership – and wasn’t that the goal to start with?[*] I offer two equations as a summary:

I. {Ideal (SMART) Measure} minus {Internalized Meaning} =  {Sexy Research, but No Signal for Change Management}

II. {Imperfect Internalized Measure} = {Sub-Optimal Signal for Change Management} plus {Potential for Improvement over Time}

So, II might be more conducive to guiding change, even if--and that is a clear risk--"sub-optimal signals" carry the risk of being misleading. Hence the need for solid M&E professionals to help us manage this risk.

In conclusion, why you wanted to measure ownership, in an imperfect ‘rugged’ world, must lead you to choose to lean toward the right hand of the spectrum, to respect the process, and through that process improve the quality of your measures, rather than aim for an illusory perfect measurement in search of meaning and later begging for "data use".

I do not dismiss the importance of finding good measures of institutionalization and other elements afferent to ownership, or the risk inherent to "imperfect measures". And my argument is not about being satisfied with qualitative stakeholders’ perceptions. It is about the process we need to use to produce metrics and what must come first. 

Let me summarize this complexity in one sentence: You do not measure ownership without the owners.

Guess, it is simple enough after all

We -- and this "we" must be a true "we" -- have our work cut out.


[*] For complexity geeks, the reason for which it is easier to go from B7 to B5 than from B5 to B7 is because the landscape we have drawn is not only rugged, but it is also ‘dancing’ and changing. Starting from B7 and engagement of stakeholders, you may see the landscape change so that more reliable signals can be picked up (toward B5) without losing the sense-making of stakeholders’ involvement. But if you start at B5, you may see the next peak get higher.

Last note on "dancing landscape", it seems that "country ownership" is falling off the PEPFAR lexicon... so, to be continued I guess.

Acknowledgement: I owe the concepts of rugged and dancing landscapes to Scott E Page's presentations and books.