Are Rapid Evaluations Helping African Governments Generate Quality Evidence Faster?
In 2019 Twende Mbele partners introduced rapid evaluations into the existing Twende Mbele countries’ national evaluation systems in an effort to address time lags in providing results to decision-makers. Adopting a rapid evaluation approach is envisaged to deliver timely, accurate, and insightful evaluation results, leading to enhanced use of results in decision making.
Twende Mbele partners have so far adapted a collaboratively generated rapid evaluation guideline to their country context, trained government officials in practical approaches to rapid evaluation, and supported the implementation of pilot rapid evaluations. This work has highlighted a few challenges that contest whether the public sector is ready for rapid evaluations, or if use of evidence in this form will be hindered by the same challenges as regular evaluations.
Why rapid evaluations and not traditional evaluation approaches
Rapid evaluations are intended to be used for a variety of evaluations questions, where results are needed to feed into programmes and strategies quickly (and on a smaller budget). While rapid evaluation approaches are useful for government decision making on certain evaluative questions (particularly around implementation and learning), more traditional evaluation approaches are needed for complex policies and programmes, or where there is a degree of political sensitivity and/or a need for a very rigorous understanding of impact.
Rapid evaluations can also be distinguished in terms of their intended purpose, for example, being conducted in real-time, or alongside larger evaluations, to support innovation, intervention, development and implementation.
Demand and use for rapid evaluations by African governments
Just like a more traditional evaluation approaches, weak evaluation culture hinders good governance and evidence-informed decision-making. Creating an evaluation culture requires buy-in from government ministries and agencies, to parliaments, to the grassroots level. Building and strengthening an evaluation culture should enhance the effectiveness, efficiency and accountability in the management of development policies and programmes. There must be a steady supply of high quality evaluations (rapid or otherwise), and demand for these evaluations in order to ensure their use.
For this reason, with the help of other partners, Twende Mbele continues to advocate for the use and benefits of rapid evaluations and evidence use as a whole.
Benin, South Africa and Ghana have conducted rapid evaluations with inputs from the Twende Mbele toolkit. These rapid evaluations ranged from 6 to 8 months to complete, and although the time taken to complete these evaluations are longer than envisioned in the rapid evaluations toolkit and guidelines, this is still a shorter turnaround time as compared to traditional evaluations. More positively, majority of the evaluation recommendations have fed into improvement plans and change in programs.
Quality of rapid evaluations by African governments
Twende Mbele has also supported a Masters research project intended to clarify our understanding of the use of rapid evaluations, the conditions under which they are commissioned, the level of research rigour they carry, and the reason for commissioning them. The results of that study found that:
- Commissioners working in Sub-Saharan Africa commission most of the rapid studies, confirming that evaluation evidence tend to follow development aid activity globally.
- Rapid evaluations are found to be more narrowed in focus, generally studying single outcome interventions or phenomena.
- he study found that over 75% of the rapid studies are either intended as a situation analysis or a baseline study, and rarely as an evaluation of programme effect or impact evaluations.
- The paper has also found that externally conducted rapid studies, rapid studies with high levels of participation by beneficiaries, and rapid evaluation intended to make a “go/no go” decision; all take at least three times longer to conduct (measured in weeks).
- The research also found a positive correlation between research rigour and duration of rapid evaluations.
In all, the research paints a picture of rapid evaluations as a tool requiring better planning and implementation to ensure the quality and the timeliness of the evaluation. The lack thereof, could impede on its rigour and robustness. As such, more work is needed to improve their quality and robustness.
Conclusion
There is no need for African governments to have to choose between the use of rapid evaluation and more traditional evaluation – as rapid evaluations can be used in conjunction with more traditional evaluations- more time allows for more data points, more interviews, more literature, and more time in ensuring that the process is good quality, involves stakeholders etc. And more money would ensure access to resources. A rapid evaluation approach can help shorten some of these processes, especially where they are planned for.
The Growing Practice of Rapid Evaluations in Africa – Insights from Three Countries
In a 2017-19 research study by Twende Mbele to assess the state of M&E culture in five participating countries which included, Benin, South Africa, Uganda, Kenya and Ghana a question regarding the ‘[timeliness of] information provided to decision-makers’ was asked to more than 462 managers.
According to a respondent, information needed to make key decisions does not make it to them on time and when it does, it is not relevant anymore. This suggested that there was a problem with key evidence being available when needed to make decisions. This and other reasons led Twende Mbele to begin an initiative to look at rapid evaluations to plug this gap.
Rapid evaluations are intended to reduce the costs of evaluation projects and the time they take (DPME, 2020). These are evaluations which can produce a result that can feed into policy and practice quickly, but yet is sufficiently robust to provide good guidance for decision-making.
They address the need to quickly assess policy/programme/strategy/function delivery, and establish the main performance data, with main recommendations for improvements (Hercules, 2019). And they also help to understand and learn from what works, what doesn’t, when and for whom.
The last couple of years has seen the advancement of methods for doing Rapid Evaluations, many African countries have taken up the opportunities to conduct this type of evaluation for quicker results. This blog gives a short insight and update of three Twende Mbele country governments in doing the type of evaluation.
At the start of 2019, Twende Mbele designed a Rapid Evaluation Toolkit to provide a framework to think about rapid evaluations, and example questions, indicators and tools to do rapid evaluations. This toolkit is intended for use by officials in evaluations, research and program delivery working in all spheres of government.
In South Africa, the Department of Planning Monitoring and Evaluation (DPME) introduced a Rapid Evaluation Guideline in May 2020. The guideline was developed collaboratively between the DPME, the Western Cape Department of the Premier (DoTP) and the Twende Mbele. The Guideline is an introduction to doing rapid evaluations, either conducted internally or externally. It was designed to provide guidance for implementing rapid evaluations in a government context.
Borrowing from the South African Rapid Evaluation Guideline, Benin adapted the guidelines to its existing national guidelines. A training of National Evaluation System (NES) actors took place in June 2020, with the aim of developing the capacities of national actors in this new form of evaluation for use in the event of an emergency, or as part of a preliminary analysis to help determine priorities, identify emerging problems and trends, and enable decision-making to support adjustments to an intervention. With that said, the Bureau of Evaluation of Public policy and Analysis of Government Action BEPPAAG will be undertaking a Rapid Evaluation of the recent effects of COVID-19 on the informal sector.
In Ghana, the Ministry of Monitoring & Evaluation conducted their first rapid evaluation with IDInsight in 2019. This was the Rapid Evaluation of the One Village One Dam (1V1D) project, an intervention which seeks to increase access to reliable source of water for livestock watering, domestic activities and dry season farming as a means of contributing to poverty eradication and addressing the various forms of inequalities with particular emphasis on rural and deprived communities.
The aim of the Rapid Evaluation was to assess the progress of implementation, emerging outcomes and potential impacts of the 1V1D intervention. It also sought to document emerging issues and challenges that will require immediate attention of the Cabinet, Ministry for Special Development Initiative (MSDI) and other stakeholders.
For critical decision-making, Rapid Evaluation have been used in response to unplanned or planned actions by governments for urgent information needs. Inevitably, there is a trade-off for one to use Rapid Evaluation instead of more rigorous evaluation in the public sector. While there is potential for a greater use of findings when they are timely, a longer evaluation allows for more data points, deeper investigation of the literature, etc.
Second annual gLocal Evaluation Week – 2020
With the ten-year countdown to the Sustainable Development Goals underway, Systems to evaluate the impact of policies and monitor their progress are more important than ever. To sustain the momentum of global efforts to promote monitoring and evaluation capacity the CLEAR Initiative will convene the second annual gLOCAL Evaluation Week from June 1nd to 5th of this year. To take part in this year’s gLOCAL Evaluation Week, simply register and submit your proposal. Applications close 6th March 2020. Click here to register….
The Republic of Ghana Public Sector and M&E Culture
In 2017 the Government of Ghana added a new tool to its development process by establishing the ministries for Planning (MoP) and for Monitoring and Evaluation (MoME). The MoME and the NDPC (National Development and Planning Commission) and the local M&E Association – the Ghana National Monitoring and Evaluation Forum (GMEF) – have developed a National Monitoring and Evaluation Policy. Once cabinet approves the Policy, it will be an important instrument in fostering the culture and practice of evidence-based decision-making in the public sector. The institutionalisation of M&E calls for better planning, use of resources, implementation and learning from outcomes about what worked and what did not.
With the establishment of the MoME and the recent launch of GIMPA’s (Ghana Institute for Management and Public Administration) Masters programme for evaluators, a better coordinated ecosystem for evaluation appears to be forming. However, the government system still faces many challenges, such as:
- weak M&E capacities,
- low demand for, and utilisation of M&E results, limited resources and budgetary allocations for M&E,
- non-compliance with M&E reporting timelines and formats by MDAs (Ministries, Departments and Agencies) /MMDAs (Metropolitans, Municipalities, Districts and Agencies)
- inconsistent data quality,
- data gaps, and
- limited management information systems.
Given that building evaluative thinking and systems is a relatively new endeavour, research into the current M&E culture in the public sector would prove useful. It is for this reason that the Ghanaian government partnered with Twende Mbele to undertake a study to establish a baseline of the M&E culture in the public sector that can be used to measure changes in practices and attitudes over the coming years. The study included interviewing 43 senior management officials from 14 ministries and two agencies using a formulated survey. The results below are based on the Report on the Ghana M&E Culture Baseline Survey.
All (100%) respondents to the survey indicated that evaluation reports are not structured to hide results. Futhermore, most (95%) reported that evaluation results of poor performance are not ignored and 92.50% that senior management does not reject evaluation reports with findings of poor performance.
The MoME is currently working on frameworks and policies to standardise and improve use of M&E across government. This includes the development of a mechanism to obtain information to understand the cause of poor performance. The above suggests a positive management attitude and environment within which to practice M&E in the public service.
According to survey results, recommendations are implemented and learning outcomes are documented and used to improve future results – 92.50% of respondents stated always. The implementation of performance management and performance contracts with managers of the ministries has yielded some results as performance measurement and results are now a basic requirements by Senior level Managers, Ministers and Cabinet for decision-making. There was however, a perception that the officers responsible for poor performance are not sanctioned, (32/38 of respondants stated never), which needs to be addressed in order to motivate staff. The report therefore recommended that a rigorous sanction and rewards regime must be implemented to regulate staff performance in the Ghanaian public service.
An enduring M&E system cannot be developed overnight. However, the Ghanaian government has proven that it is committed to pursuing a desired change based on the work of existing institutions including NDPC, MoME and GIMPA. It has been a steady process and one of the building blocks required was behaviour change. There is also the need for a paradigm shift towards the incorporation of M&E into the routine work of public service staff in addition to deepening M&E processes by which public service activities are conducted in order to build a stronger, learning-focused M&E Culture in Ghana.
By: Rendani Manugu
Twende Mbele, South Africa, Country Coordinator
News from our collaborating partner country-Ghana
By Dr Nana Opare-Djan
Introduction
Over the past year the M&E landscape in Ghana has seen tremendous measures towards institutionalising development evaluation in the entire public sector. Contributions from several stakeholders at different fora, held across the country, gave indications of the high degree of interest in M&E especially the need in drafting of a National Evaluation Policy (NEP) to guide the conduct of evaluation for evidence based decision making at all levels of government.
M&E architecture in Ghana – NDPC/Ministry of Planning and Ministry of M&E
Given the mandate under articles 86 and 87 of the 1992 Constitution, National Development Planning Commission (NDPC) continues with the preparation of the Annual Progress Report (APR) within the framework of the Cross-Sectoral Planning Groups (CSPGs). The process involved the engagement of technical experts with the relevant background and knowledge to review, analyse, and report on progress in the implementation of policies, strategies and programmes using agreed sets of national indicators. The Ministries of Planning, Monitoring & Evaluation (MME) has been created by the amendment of the Civil Service Act, (PNDCL 327) with an Executive Instrument E.I 38 to support government results delivery.
Official Launch of the Postgraduate Diploma in Monitoring and Evaluation Programme
The Center for Learning on Evaluation and Results (CLEAR) at the Ghana Institute of Management has been playing a pioneering role in professionalization of Monitoring and Evaluation in the West Africa Sub region. As one of our key partner institutions in the M&E professionalization journey, we successfully hosted of our first official Postgraduate Diploma in Monitoring and Evaluation Programme as of February 2018.
National Public Sector Reform Strategy (NPSRS), 2018-2023
NPSRS is a five (5)-year project conceptualized to improve public sector performance, especially the delivery of services to citizens and the private sector. A stakeholders’ consultation meeting was held to finalize work on the Public Sector for Results Project (PSRRP) 2018-2023. The objectives of the meeting were to review and finalise project objectives, scope, design features and results framework with key public sector stakeholders; review the proposed components of the PSRRP and agree on details of coverage; discuss financial and procurement arrangements, including necessary assessments; and to define and agree on a preparation programme, key dates, and the budget.
National Evaluation Policy (NEP)
The Steering Committee members however met recently to update the roadmap for drafting the national evaluation policy. The committee recognized the need to have other experts to comment on the draft NEP and provide policy guidance and identify lessons learnt from other contexts for inclusion. A Reference Group responsible for providing the technical and strategic guidance is to be established to support the Steering Committee.
CreativeBox
