Planning evaluation

Purpose of the evaluation

The initial question is always why an evaluation is to be carried out in the first place, who has what interests in the evaluation and for what purpose the results are to be used. An evaluation may have various ends and may serve to:

• establish the bases for decision-making with respect to the continuation of the project
• legitimate the appropriation of public funds
• further develop a project or improve strategies and measures.

An evaluation is planned in the conceptual phase of a project. If an evaluation is not considered until the end of the project, it is usually too late: the opportunity to obtain key data has been missed, the resources for the evaluation are lacking and/or it is already too late to learn from the evaluation during the implementation phase.

For larger evaluation projects and evaluations that are to be outsourced to external organisations, it is recommended that a specific evaluation plan be developed for this purpose (please also refer to the Evaluation Plan Template). The following aspects should be addressed:

Subject of evaluation

Once the purpose and interests of the evaluation have been determined, the scope of the evaluation should be defined. Which aspects of the project are to be assessed, where should the focus lie and what should be disregarded?

Forms of evaluation

Evaluation projects are primarily distinguished by whether they assess the effects of an entire project at the end (summative evaluation) or are carried out concomitantly in order to optimise the ongoing processes and outcomes (formative evaluation). Formative and summative evaluations are often combined and carried out in hybrid form.

	Formative evaluation	Summative evaluation
Self-evaluation	Systematic learning process implemented independently by the project team	Final self-assessment of the project
Internal third-party evaluation	Supervision of the learning process by evaluators from within the organisation	Final assessment of the project by evaluators from within the organisation
External third-party evaluation	Supervision of the learning process by an external evaluation organisation	Final assessment of the project by an external evaluation organisation

The advantages of self-evaluations include internal knowledge of the project, practical relevance and an increased willingness to learn and responsibility for the results on the part of participants. Self-evaluations are often used formatively and serve to establish a culture of learning. Their disadvantages may include a lack of distance from the evaluation object or a lack of methodological expertise. Self-evaluations also entail the risk of reinforcing operational blind spots. Third-party evaluations offer an external, impartial review and external evaluators usually have proven methodological skills. External evaluations have a high level of credibility and are often used to legitimate the appropriation of funds. One of their disadvantages may be high external costs. Hybrid forms (hybrid evaluations, participative evaluations) can combine the advantages of internal self-evaluations and external third-party evaluations, supplementing insider knowledge with a fresh view from the outside.

Evaluation questions

Evaluation questions are formulated based on the purpose of the evaluation and the project’s goals, and this is best done in collaboration with key players in the project. If the project’s objectives or evaluation questions are still too vague, indicators are defined that are intended to specify the objectives of the project and questions. How can you tell when a specific project objective has been attained, and what do you have to be aware of in order to provide well-founded answers to the evaluation questions?

Evaluation methods

The project objectives, evaluation questions and indicators are used to decide which methods of collecting data will be used to collect what information, from whom or from where. At the same time, a data evaluation method must also be defined. The evaluation chart serves to define the corresponding evaluation methods, responsible individuals and time schedule for the project objectives, indicators and evaluation questions. Certain evaluation questions may pertain to individual project phases and are answered at a certain milestone, while others may encompass the entire duration of the project.

Organisational questions

The ultimate aim is to define the temporal and financial scope of the evaluation and to delegate responsibilities. If an evaluation mandate is to be awarded externally, it must be decided which methodological, thematic and skills responsibilities are required for the evaluation and which evaluation organisations would be suitable to take on the job. You should allocate 10-15% of the project's budget to the evaluation. The WHO (1998) recommends that at least 10% of a project's budget should be reserved for evaluation.

Dissemination and valorisation of results

Ultimately, the goal is to determine the expected evaluation results and how they are to be disseminated. Are interim and final evaluation reports desired? In what languages? How should summaries and presentations be created and should the results be scientifically published, if necessary? Who else should be able to benefit from the evaluation results and what options are there for processing and communicating the results in an attractive manner (e.g. guidelines, posters, films, workshops, etc.)?

Bibliographical references

• Nutbeam, D. & Bauman, A. (2006). Evaluation in a nutshell. A practical guide to the evaluation of health promotion programs. New York: McGraw-Hill
• WHO (1998). Health promotion evaluation. Recommendations to policy-makers. Report of the WHO European Working Group on Health Promotion Evaluation. Copenhagen: Regional office for Europe. Link/Download
• Van Marris B. & King, B. (2007). Evaluating health promotion programs. The Health Communication Unit, University of Toronto. Link/Download