Rapid Evaluations in Uganda: The view of a Team Leader
Introduction:
I was a team leader on two rapid evaluations in Uganda. These evaluations included:
- assessing the extent to which the use of the remitted 20% of national park entry fees has been effective in improving the livelihood of the communities in Local Government districts surrounding the wildlife protected areas;
- exploring the challenges in the Local Government staffing while focusing on the filling of critical positions in the Local Government structures in Uganda.
This blog highlights my experiences of successfully undertaking two rapid evaluations, paying special attention to: benefits of the use of a hybrid/facilitated model for doing rapid evaluation; developing rapid evaluation topics; ensuring buy-in for rapid evaluation and use; and lastly, ensuring the quality assurance of the rapid evaluation in Uganda.
Benefits of the use of a hybrid model for doing rapid evaluations
As outlined in the Rapid Evaluation Guidelines (add hyperlink here) there are a few options for governments wanting to start doing rapid evaluations. In Uganda’s case, they have moderate team capacity and wanted to use an external facilitator to assist the internal team with the technical aspects of the evaluations. This is what we call a ‘hybrid model’.
There have notable benefits along the process of using this capacity develop approach to undertaking rapid evaluations. Some of these include;
i) a mix of ideas and expertise from the mainstream government technical teams and the independent consultant, which is a good practice in evaluations;
ii) a quick consensus on the clarity of the study scope, methodologies and expected evaluation results;
iii) knowledge sharing and transfer is quite evident;
iv) an increasing impetus for buy-in, internal clearances, uptake of the evaluation findings to inform policy change as well as improve implementation of government programmes/projects.
Developing rapid evaluation topic
Developing rapid evaluation topics and questions requires demand-driven approaches. In most cases, it begins with identification of the knowledge gap/evaluation problem that seeks for quick solutions to better improve service delivery. These gaps are often identified through quarterly, semi-annual and annual reviews of various government programmes and projects, or from Cabinet and Ministers. The Office of the Prime Minister is then tasked with the process of drafting the terms of reference (ToR), securing funding, setting-up internal and/or external evaluation teams including the quality assurance procedures.
In this case, the internal team had developed selected the evaluation topics through consultation with relevant departments and ministers, and designed the evaluations during a Twende Mbele-led training on how to do rapid evaluations. As the team leader, my job was to ensure the design was feasible and rigorous, and that it could be done within the desired timelines.
Ensuring rapid evaluation buy-in and use
Throughout the rapid evaluation process, it is absolutely necessary that the commissioners seek clearance from both the technical and policy organs of government. All the processes from evaluation design to dissemination should include participation of the relevant stakeholders/consumers of the evaluation findings. It is important that several forms of disseminations are conducted such as reports, policy briefs, press conferences, symposia, website blogs, etc.
There should be follow-up meetings after the rapid evaluation reports disseminations to ensure that there is not only uptake but also use of the evaluation results to improve service delivery. As a team leader, I was responsible for ensuring there is a plan for dissemination and for quality assuring some of the written outputs done by the team.
Quality assurance
Ensuring high quality rapid evaluations is about managing the risks and errors along the processes. A clear multi-sectoral quality assurance framework was utilised to keep track of the evaluation processes. In Uganda, the quality assurance mechanisms are premised on the national M&E strategy framework of a country’s M&E eco-system. Within a quality assurance framework are principles and standards to follow in conducting rapid evaluations. The rapid evaluations in Uganda are effectively coordinated by the national M&E multi-sectoral working group guided by the national M&E strategy. In some cases, reference groups are set up to provide unique technical input to enrich the evaluation processes and findings.
Conclusion
As we can see, there are many elements that come together to ensure the success of undertaking a rapid evaluation – many of which are not dissimilar to traditional evaluations. In this blog I have highlighted some of these elements which was critical in the process of Uganda successfully doing two rapid evaluations utilising a facilitated/hybrid model.
Are Rapid Evaluations Helping African Governments Generate Quality Evidence Faster?
In 2019 Twende Mbele partners introduced rapid evaluations into the existing Twende Mbele countries’ national evaluation systems in an effort to address time lags in providing results to decision-makers. Adopting a rapid evaluation approach is envisaged to deliver timely, accurate, and insightful evaluation results, leading to enhanced use of results in decision making.
Twende Mbele partners have so far adapted a collaboratively generated rapid evaluation guideline to their country context, trained government officials in practical approaches to rapid evaluation, and supported the implementation of pilot rapid evaluations. This work has highlighted a few challenges that contest whether the public sector is ready for rapid evaluations, or if use of evidence in this form will be hindered by the same challenges as regular evaluations.
Why rapid evaluations and not traditional evaluation approaches
Rapid evaluations are intended to be used for a variety of evaluations questions, where results are needed to feed into programmes and strategies quickly (and on a smaller budget). While rapid evaluation approaches are useful for government decision making on certain evaluative questions (particularly around implementation and learning), more traditional evaluation approaches are needed for complex policies and programmes, or where there is a degree of political sensitivity and/or a need for a very rigorous understanding of impact.
Rapid evaluations can also be distinguished in terms of their intended purpose, for example, being conducted in real-time, or alongside larger evaluations, to support innovation, intervention, development and implementation.
Demand and use for rapid evaluations by African governments
Just like a more traditional evaluation approaches, weak evaluation culture hinders good governance and evidence-informed decision-making. Creating an evaluation culture requires buy-in from government ministries and agencies, to parliaments, to the grassroots level. Building and strengthening an evaluation culture should enhance the effectiveness, efficiency and accountability in the management of development policies and programmes. There must be a steady supply of high quality evaluations (rapid or otherwise), and demand for these evaluations in order to ensure their use.
For this reason, with the help of other partners, Twende Mbele continues to advocate for the use and benefits of rapid evaluations and evidence use as a whole.
Benin, South Africa and Ghana have conducted rapid evaluations with inputs from the Twende Mbele toolkit. These rapid evaluations ranged from 6 to 8 months to complete, and although the time taken to complete these evaluations are longer than envisioned in the rapid evaluations toolkit and guidelines, this is still a shorter turnaround time as compared to traditional evaluations. More positively, majority of the evaluation recommendations have fed into improvement plans and change in programs.
Quality of rapid evaluations by African governments
Twende Mbele has also supported a Masters research project intended to clarify our understanding of the use of rapid evaluations, the conditions under which they are commissioned, the level of research rigour they carry, and the reason for commissioning them. The results of that study found that:
- Commissioners working in Sub-Saharan Africa commission most of the rapid studies, confirming that evaluation evidence tend to follow development aid activity globally.
- Rapid evaluations are found to be more narrowed in focus, generally studying single outcome interventions or phenomena.
- he study found that over 75% of the rapid studies are either intended as a situation analysis or a baseline study, and rarely as an evaluation of programme effect or impact evaluations.
- The paper has also found that externally conducted rapid studies, rapid studies with high levels of participation by beneficiaries, and rapid evaluation intended to make a “go/no go” decision; all take at least three times longer to conduct (measured in weeks).
- The research also found a positive correlation between research rigour and duration of rapid evaluations.
In all, the research paints a picture of rapid evaluations as a tool requiring better planning and implementation to ensure the quality and the timeliness of the evaluation. The lack thereof, could impede on its rigour and robustness. As such, more work is needed to improve their quality and robustness.
Conclusion
There is no need for African governments to have to choose between the use of rapid evaluation and more traditional evaluation – as rapid evaluations can be used in conjunction with more traditional evaluations- more time allows for more data points, more interviews, more literature, and more time in ensuring that the process is good quality, involves stakeholders etc. And more money would ensure access to resources. A rapid evaluation approach can help shorten some of these processes, especially where they are planned for.
Programme Manager Needed
Programme Manager Needed
We are looking for a motivated programme manager with experience in monitoring and evaluation capacity development to lead our peer learning agenda and oversee the sustainability of the program.
Application Deadline: 30th May 2022
Location: Remote/South Africa
link: Full Terms of Reference
Send applications to: [email protected]
gLocal Evaluation Week 2022
How Can Datafication Augment M&E in the African Public Sector
Join this gLocal Evaluation Week discussion and learnhow data collection, analysis and use is influencing the M&E field.
Register for the webinar by pressing: Register Here
Resource Mobilisation Consultant Needed
Short-term Resource Mobilisation Consultant Needed
Twende Mbele is seeking the services of a qualified consultant or consultants to develop a comprehensive resource mobilisation strategy that will take into account the current global donor landscape, identify new actors, explore new funding modalities, and outline a strategic approach to leveraging existing and new partnerships, as well as make recommendations on how to diversify our resource base.
Application Deadline: 25 May 2022
Location: Remote/South Africa
link:
Full Terms of Reference
Send applications to: [email protected]
CreativeBox
