Working with Assumptions in International Development Program Evaluation (2^nd Ed., 2nd ed. 2020) With a Foreword by Michael Bamberger

This book discusses the crucial place that assumptions hold in conceptualizing, implementing, and evaluating development programs. It suggests simple ways for stakeholders and evaluators to 1) examine their assumptions about program theory and environmental conditions and 2) develop and carry out effective program monitoring and evaluation in light of those assumptions. A survey of evaluators from an international development agency reviewed the state of practice on assumptions-aware evaluation. This 2^nd edition has been updated with further illustrations, case studies, and frameworks that have been researched and tested in the years since the first edition.

Regardless of geography or goal, development programs and policies are fueled by a complex network of implicit ideas. Stakeholders may hold assumptions about purposes, outcomes, methodology, and the value of project evaluation and evaluators?which may or may not be shared by the evaluators.  A major barrier to viable program evaluations is that development programs are based on assumptions that often are not well articulated. In designing programs, stakeholders often lack clear outlines for how implemented interventions will bring desired changes. This lack of clarity masks critical risks to program success and makes it challenging to evaluate such programs. Methods that have attempted to address this dilemma have been popularized as theory of change or other theory?based approaches. Often, however, theory-based methods do not sufficiently clarify how program managers or evaluators should work with the assumptions inherent in the connections between the steps. The critical examination of assumptions in evaluation is essential for effective evaluations and evaluative thinking.

"How does one think evaluatively? It all begins with assumptions. Systematically articulating, examining, and testing assumptions is the foundation of evaluative thinking?  This book, more than any other, explains how to build a strong foundation for effective interventions and useful evaluation by rigorously working with assumptions." 
?Michael Quinn Patton, PhD. Author of Utilization-Focused Evaluation and co-editor of THOUGHTWORK: Thinking, Action, and the Fate of the World, USA.

"This updated edition presents us with a new opportunity to delve into both the theoretical and practical aspects of paradigmatic, prescriptive, and causal assumptions. We need to learn, and apply these insights with the deep attention they deserve."  
?Zenda Ofir, PhD. Independent Evaluator, Richard von Weizsäcker Fellow, Robert Bosch Academy,    Berlin, Germany. Honorary Professor, School of Public Leadership, Stellenbosch University, South Africa.

?This thought-provoking book explains why assumptions are an essential condition within the theories and methodologies of evaluation; and how assumptions influence the ways that evaluators approach their work?It will enrich the ways that evaluators develop their models, devise their methodologies, interpret their data, and interact with their stakeholders.?
?Jonny Morell, Ph.D., President, 4.669? Evaluation and Planning, Editor Emeritus, Evaluation and Program Planning

Part I Designing and Evaluating Complex Development Programs

Chapter 2 Attributes of Complex Development Programs

Abstract

This chapter draws on complexity literature to outline attributes of complex development programs. Most attributes of complexity—such as dynamism, emergence, adaptation, nonlinearity, and scale—are likely to facilitate program success, but make it difficult to measure that success. In trying to measure success of complex programs, evaluators and program stakeholders often assume the reverse—that these programs are simple, linear, and predictable. An even greater problem is that these assumptions, huge as they are, are taken for granted (implicit); neither explicated nor tested. In the second edition, the chapter will be revised to address recent applications of complexity-aware program evaluation. 

Chapter 3 Designing Complex Development Programs

Abstract

Approaches to designing development programs vary. Some emphasize stakeholder involvement; others focus on a program’s environment; still others emphasize the sequence of change depicted in program results (change frameworks). This chapter examines the use of such change frameworks as the Logical framework approach (LFA), Theory of change approach (ToCA), and Participatory impact pathways approach (PIPA) for designing complex development programs and the extent to which such frameworks enable stakeholders to explicate and question implicit assumptions. In the second edition, this chapter will be revised to discuss applications of change frameworks to examine assumptions of programs. 

Chapter 4 Evaluating Complex Development Programs

Abstract

Evaluators express preferences methods over others. This chapter highlights the debate and assumptions underlying these preferences. There are debates on which methods represent the ‘‘gold standard’’ for evaluation. The key point discussed in this chapter is that the ‘‘gold standard’’ for evaluation methodology is Appropriateness to evaluation questions and the contextual realities of complex programs. In the second edition, this chapter will be revised to address current debates on methodology for evaluation of complex development programs.

Part II Evaluation Theory and Assumptions

Chapter 5

Theory in Evaluation

Abstract Distinctions are drawn among social science theory; evaluation theory; evaluators’ theory; and program theory. Similarly, the distinguishing characteristics that mark differences between results chains and theories of change are discussed. The chapter underlines the concept that logic models are incomplete if they do not spell out the assumptions (implicit/or behind-the-scenes concepts, conditions, and qualifications) within the linkages or arrows in a results chain, which explain why result A is expected to lead to result B and so on. In the second edition, the role of theory in comprehension and measurement of complex reality is discussed. I will argue that it is impossible, as evaluators, not to simplify our view of the complex world if we are to act. This constraint will always be with us. Simplification, with the use of models is a common resort and intended to assist designers of programs and evaluations to comprehend the complex reality. Both intentionally and unintentionally, models account only for some of the reality. Some model elements may be justified by data; others are mere leaps of faith. Models also capture ignorance because there is much we might put in, but we do not know enough to even think about the possibilities. It is common that elements left out of the model ultimately become of critical importance-the assumptions. It is sometimes (actually often, maybe most of the time) rational and adaptive to ignore assumptions even when we are making them.

 

Chapter 6 What are Assumptions?

Abstract

Although the pertinence of implicit assumptions is underlined in theory based evaluations, the nature of these assumptions has been rarely discussed. To understand the nature of underlying assumptions, it is necessary to review the notion of assumptions in general and its remarkable role in the generation of knowledge. This chapter discusses the nature of assumptions and groups in 10 categories of assumptions according to Brookfield’s (Becoming a critically reflective teacher. San Francisco, CA: Jossey-Bass, 1995) typology: Paradigmatic, Prescriptive and Causal assumptions and according to the degree of articulation. In the second edition, this chapter will be updated with case study data on the current practice in examining assumptions-which assumptions are commonly examined.

 

Chapter 7 Why are Assumptions Important?

Abstract

Assumptions are the foci for any theory and thus any paradigm. It is also important that assumptions are made explicit, and that the number of assumptions is sufficient to describe the phenomenon at hand. Explication of assumptions is even more crucial in research methods used to test the theories. As Mitroff and Bonoma (Evaluation Quarterly 2(2):235–260, 1978) have eloquently put it: ‘‘the power of an experiment is only as strong as the clarity of the basic assumptions which underlie it. Such assumptions not only underlie laboratory experimentation but social evaluation research as well.’’ A scale of articulation of assumptions is represented: (a) Very ambiguously tacit assumptions held in divergent beliefs; (b) Tacit but more obvious assumptions-where parties more or less have shared although unexpressed perceptions or beliefs e.g., legal assumptions in litigation; (c) Informally, explicit assumptions e.g., indirectly expressed or inherent in shared stories, norms, un-codified symbols, and myths; (d) Assumptions that are made explicit. E.g. in theories and models, policies and programs, research and methodology but untested; and (e) Explicit and tested assumptions. In the second edition, this chapter will be updated with case study data on the current practice in examining assumptions-why are assumptions thought to be very important.

 

 

Part III Explicating Tacit Program Assumptions

Chapter 8 Normative and Diagnostic Assumptions

Abstract

Normative assumptions pertain to the expectation that something must change; that the existing condition is less than ideal, hence the design and delivery of an intervention. Diagnostic assumptions are stated as stakeholders’ expectations or beliefs of the major and minor causes of core problems. Since the intervention to address a problem is based on the causes of that problem, diagnostic assumptions are crucial to a normative theory and need to be examined from design, implementation, and evaluation perspectives. This chapter appraises the use of the policy scientific approach in explicating diagnostic assumptions. An Alternative Causes Approach is proposed. In the second edition, this chapter will be updated with case study data on the current practice in examining normative and diagnostic assumptions.

 

Chapter 9 Prescriptive Assumptions

Abstract

Prescriptive assumptions relate to the intervention or strategy devised for the problem or to reach a stated objective, which represents stakeholders’ beliefs of what could be the best ways to address the problem or need. The use of goal evaluation and strategy assessment for explicating prescriptive assumptions is appraised. The alternative strategies approach is proposed. In the second edition, this chapter will be updated with case study data on the current practice in examining prescriptive assumptions. In the second edition, this chapter will be updated with case study data on the current practice in examining prescriptive assumptions.

 

 

Chapter 10 Transformation Assumptions

Abstract

While prescriptive assumptions are related to strategies (and alternatives) devised to address a problem, transformational assumptions relate to how the immediate results of a strategy program or intervention (outputs) are expected to lead to long-term desired changes. The use of Elicitation, Program theory matrix and the theory of change Approach for explicitizing transformational assumptions is appraised. An integrative approach is proposed. In the second edition, this chapter will be updated with case study data on the current practice in examining transformation assumptions.

 

 

Part IV Working with assumptions in program evaluation

Chapter 11 Evaluating Assumptions

Abstract

Examining the assumptions that hold a program theory together is a vital part of evaluating program outcomes. Examining implicit or explicit program assumptions facilitates understanding of program results-both intended and unintended. Evaluation approaches for testing program assumptions are outlined. The best place to start integrating assumptions in an evaluation is at the conceptualizing stage, when evaluation questions are being formulated, not in data collection or methodologies. Tools are but a servant of methods and methods a servant of questions, which should be the servant of objectives and/or purpose. Ideally, by framing the questions well, the methods, tools, and data will produce highly useful answers and solutions. But examining assumptions is without doubt a necessary element in the process. In the second edition, this chapter will be updated with case study data on the current practice in examining evaluating assumptions.

Chapter 12 Challenges of Explicating Assumptions

Abstract

The process of explicating assumptions requires an analytical stance that is different from the empathetic, responsive, and intuitive stance of many stakeholders (Weiss New Approaches to Evaluating Community Initiatives, Theory, Measurement and Analysis, vol 2, pp. 65–92, 1995). Because developing and questioning the underlying assumptions of a program’s theory can be threatening to participants, it is important that the process be done in a sensitive and collaborative way, so that it strengthens the program without dampening enthusiasm or weakening stakeholder commitment. Explicating implicit assumptions is something stakeholders may be uncomfortable with for different reasons. In the second edition, this chapter will be updated with case study data on the challenges and strategies for examining assumptions.