Once the evaluation questions have been defined, the investigation methods must be specified in order to obtain the desired answers.
In the social sciences, quantity and quality-based research principles have been established which differ in the focus of their findings and in their methods. As the terms suggest, a quantity-based research design focuses on quantifying phenomena, or measuring their quantities. A quality-based design looks for different opinions, perspectives and backgrounds. Although this is often suggested, individual data collection methods and their corresponding collection tools cannot be clearly assigned to one of these categories. Most data collection tools can be used both quantitatively and qualitatively, depending on their degree of open-endedness. For example, a questionnaire may contain quantitative, closed-ended questions in addition to open-ended questions that can be evaluated qualitatively. On the other hand, it is possible to integrate quantitative elements into interviews, e.g. if an interviewee is asked to rate something on a scale of 1-10, which can then be numerically evaluated.
In quantitative methods, quantifiable characteristics are measured and facts are represented numerically. Results are investigated by means of statistical relationships and calculations. Standardised questionnaires and observations are based on previous knowledge and focus on very specific aspects that are being broadly studied, reviewed, or for which estimates are to be collected.
Quantitative methods focus on predetermined questions and therefore provide a limited point of view. New aspects that are not included in the answer categories provided are often unaccounted for as a result.
A qualitative approach is particularly efficient for a more thorough investigation of phenomena which are complex or of which there is little previous knowledge. Open-ended interviews or observations can capture key aspects that would remain undiscovered with a highly standardised method and, specifically, with closed-ended questions. Qualitative methods have their own set of quality criteria that are distinct from the classic quality criteria for quantitative investigation, e.g. intersubjective traceability, subject suitability, scientific foundation, reflexivity or context relevance (Steinke 1999, Studer 2011).
Qualitative methods are often used when the subject of the investigation is relatively new or in order to investigate the field of study or question and to develop hypotheses. Quantitative methods tend to be used to reveal the frequency or distribution of certain phenomena or to test existing hypotheses. Qualitative and quantitative methods are often combined. If a certain subject is examined from different methodological perspectives at the same time, e.g. investigated with multiple qualitative and quantitative methods, this is referred to as method triangulation (Flick 2004).
There are a number of methods and corresponding tools that may be used in evaluations. The best method in each case depends on the question, environment, target group, financial and personnel resources and other factors.
|
Evaluation methods |
Remarks |
|
Document analyses |
Existing documents (reports, concepts, graphic material, videos, etc.) may be relied upon for individual questions or aspects, eliminating the need for additional data collection. |
|
Written surveys (paper version) |
Written surveys are the most common data collection method when information is to be collected from medium-sized or large population groups in order to obtain a statistically representative overview of the group's responses. Questionnaires may contain both closed-ended (with predetermined answer categories) and open-ended questions. |
|
Online surveys
|
Unlike other written surveys, online surveys have the advantage of avoiding the need for subsequent electronic data entry, which saves a lot of time. However, not all people and groups can be reached this way. |
|
Telephone surveys |
Telephone surveys are becoming increasingly common. Compared to written surveys they are very time-consuming, in particular with larger population groups. However, they offer an efficient alternative for smaller groups. Unlike written surveys, they offer the convenience of being able to follow up in the event of confusion or when more information is needed. |
|
Telephone interviews |
Both quantitative and qualitative interviews can be carried out over the phone in order to delve deeper into certain questions. Compared to personal interviews, telephone interviews save travel time and costs. |
|
Personal interviews |
Personal interviews usually generate a greater sense of trust than telephone interviews and they offer additional insight into the subject’s circumstances, provided they are carried out in the field. |
|
Group surveys |
Unlike individual interviews, group surveys allow new aspects to emerge and be addressed collectively. Group interviews are therefore particularly suitable when a consensus is to be reached in assessments. However, the perspectives of individuals cannot be elaborated on to the same extent or investigated independently of the other participants. |
|
Observations |
Observations are used to assess events on site, firsthand, without them being filtered by the interviewees. Even subliminal, subconscious, inexplicit factors can be recorded in this manner. However, only aspects that are directly subject to observation can be evaluated, i.e. not someone's attitudes, for example. |
|
Measurements |
Specific aspects can be objectively measured using the appropriate tools, for example someone's height and weight can be measured to calculate their body mass index (BMI) and monitor it over time, or time can be measured, to see how long someone needs to perform specific tasks. In these cases the measuring tools would be scales, a tape measure and a clock. |
|
Tests |
Although not used as frequently, tests can serve a multitude of purposes in evaluations in order to assess intervention methods or products (e.g. pre-tests for poster campaigns, test purchases in alcohol prevention, usability tests on websites, etc.). |
In addition to these widely used data collection methods, there are a host of other methods that can also be used for evaluations (e.g. storytelling, structural constellations, experiments, auto-photography, etc.).
A well-founded choice of evaluation methods and tools guarantees that the collected data will actually provide the desired information and will be able to answer your evaluation questions. At the same time, you will avoid collecting too much data which will not (or cannot) be evaluated later.
Once you have defined the evaluation questions, check to see which data are already available and can be used to answer the questions. Allow experienced evaluation specialists to advise you in determining the evaluation methods and developing evaluation tools.