Frequently Asked Questions about Logic Models
Q Who has used or supported use of logic models?
A Program logic models have multiple sources of inspiration. One source is early systems theory and efforts to conceptualize different stages or levels of results. Independently, funders and evaluation practitioners developed various forms of logic models used extensively in the United States . Logic models were widely adopted to serve the increasing focus on accountability in public service work. Cooperative Extension agents and evaluators played an important part in the dissemination of logic models.
Another source is extensions of the Logical Framework Approach (LFA) initially developed in 1969 for the U.S. Agency for International Development by the consulting firm Practical Concepts. The LFA was adopted as a participatory planning tool by the German, Norwegian, and Danish governments. The logical framework approach has continued to evolve, with the International Service for National Agriculture Research taking the lead. For examples, see “The third generation logical framework approach: dynamic management for agricultural research projects” and “Enhancing the Impact of Agricultural Research: An Impact Pathway Perspective.”
For additional references on the development and use of logic models and logical frameworks and their use in evaluation, see selected references.
Q What forms do logic models take?
A- A program logic model, as used by NIFA, is visually represented as a graphic diagram (using boxes, links, and text), describing how the program is expected to work, what it is expected to do, with what results, under which conditions. To review a generic logic model for NIFA reporting, see this generic logic model.
Program logic models may be depicted in a wide variety of different forms, and need not be linear.
Q What are examples of different types of logic models?
A Logic models in various formats are used by many agencies, with differing programs, for different purposes.
For examples of logic models developed by NIFA for specific NIFA portfolios of research, education, and extension portfolios, see Portfolio Evaluation.
For examples of logic models developed by States, see Logic Models for 17 Issues developed by Pennsylvania State University.
Q What are the elements of a logic model?
A As used by NIFA for reporting purposes, the elements of a logic model include:
- Situation Specification of the problem, need, or situation to be changed.
- Inputs Financial, human, and material resources.
- Activities Services or functions carried out by a program (i.e., what the program does).
- Outputs Things produced in the course of services or functions.
- Outcomes to be identified at three stages,
- Changes in knowledge or learning (or short-term);
- Changes in behavior (or medium-term); and
- Changes in the conditions specified in original situation expected to occur (in the long-term).
- Assumptions The verbal statement of the program theory, ideally citing substantive theory, research, previous evaluations, or other science-based evidence, making the claim that this program could plausibly be expected to produce the expected results.
- External Factors Variables not included in the model which could help or hinder the program in producing the intended outcomes.
Q How does NIFA use logic models?
A NIFA uses logic models for four primary purposes: Planning, management, accountability, and evaluability assessment.
NIFA uses logic model development to build consensus in developing programs and portfolios, including which goals are served by which projects, what projects should be included, how program elements fit together conceptually, and how particular planned programs or portfolios might be improved by additional theory and research.
The online format for the State Plans of Work (POW) uses a logic model outline.
NIFA uses logic models in the self-assessment process. Describing what programs are perceived to be by national program leaders (NPL) and others highlights gaps between what is supposed to happen and what actually happens, and facilitates management or other changes.
Accountability and Evaluation
Accountability can be defined as the principle that individuals and organizations are responsible for their actions and can be required to explain them to others. NIFA staff are required to ensure that programs operate and resources are used consistently with agency missions, in compliance with laws and regulations, and are well-managed. Accountability systems include planning, budgeting, management, accounting, and auditing. Accountability systems assume performance can be improved, and use information and feedback to assess, revise, and improve performance.
NIFA uses logic models to depict the program theory supporting NIFA portfolios for the benefit of expert peer reviewers. NIFA uses program theory (as laid out in text describing assumptions, citations to research, previous evaluations, and past practice) in supporting and improving work done for NIFA portfolios and in responding to recommendations by peer reviewers and others.
Logic models are also useful in developing questions to be addressed through evaluation, and in identifying appropriate evaluation and data collection methods.
An evaluability assessment is a process used to determine the feasibility of a program evaluation. It also helps determine whether conducting a program evaluation will provide useful information that will help improve the management of a program and its overall performance.
Logic models may be used in thought experiments' as well as actual research, experimentation, or evaluation. For example, developing a logic model may make evaluation unnecessary, as a logic model may show that the program is so ill-conceived that more work needs to be done before the program can be implemented or if implemented, before the program is evaluated.
Q How does NIFA use logic models in evaluability assessment?
A As an example of evaluability assessment, NIFA requires logic models in the State Plans of Work (POW). NPLs review the State POWs. In doing so, they consider how the program theory depicted by the logic model may accord to actual events (results), including the sequence in which they occur AND the assessment or evaluation design to capture results. The model can be restated using a series of IF-THEN statements:
- IF the project is implemented as planned (resources provided, activities carried out as expected), THEN the expected outputs will be produced.
- IF the expected outputs are achieved, THEN the expected short-term outcomes will be realized.
- IF the expected short-term outcomes are realized, other things being equal, THEN the medium term outcomes will be realized.
- IF the expected medium term outcomes are realized, other things being equal, THEN the long-term results will be realized.
The program theory is thus tested by the correspondence between the planned results and the actual ones. Evaluability assessment is useful in considering whether it is likely or even plausible that programs can be implemented as described, will produce results if implemented, and whether they will contribute to meeting NIFA goals.
If the planned results cannot or are not realized in this sequence as expected, then:
- The program theory may have been wrong, and/or;
- The program may have been poorly designed, and/or;
- The program may have been poorly implemented, and/or;
- The evaluation design used to capture results was inappropriate (wrong sample, instruments, timing, measures, etc.), and/or;
- External events intervened and swamped the logical sequence postulated.
Replication of programs, additional observation, research, theory or evaluation of related programs can help to identify which of the above conditions apply.
For more information on program theory, attribution, and testing see the discussion paper Addressing Attribution Through Contribution Analysis: Using Performance Measures Sensibly.
For more on evaluability assessment, see the Practical Assessment, Research & Evaluation journal.