Share insight
Idea In Brief
Effective procurement involves more than just writing a good RFQ
Those commissioning evaluations must ask more fundamental questions from a higher vantage point.
When should you outsource evaluations?
External parties are sometimes best placed to deliver evaluations, or at least to lead some monitoring and evaluation activities.
Different types of evaluation provider have different strengths
It can be helpful to engage a partnership or consortium to access the strengths and mitigate the challenges associated with different types of providers.
Governments, businesses and not-for-profit organisations make considerable investments in evaluations to understand the efficiency, effectiveness, and value of their work. For example, the Australian Government spends over $50 million each year engaging external parties to conduct evaluations of their policies and programs. Getting value for money from this investment is important, both in terms of using taxpayers’ money wisely and ensuring that decision-making about policies and programs are supported by evidence.
Procurement of skilled, external evaluators is one of the main levers that organisations and governments have at their disposal to drive quality, independent evaluations. But effective procurement involves more than just writing a good request for quotation (RFQ) and carefully assessing respondents’ applications. Those commissioning evaluations, both within and outside government, must also ask more fundamental questions from a higher vantage point. What are the general circumstances in which they should outsource evaluation delivery? And, assuming that an external party is best placed to deliver an evaluation, who should they outsource to?
Nous has over 25 years’ experience working with clients to deliver evaluations across a range of sectors. This article shares some reflections about effective procurement of evaluations. We explore each of the questions set out at Figure 1.
When should you outsource evaluations?
The first step for policy makers and program managers is to decide whether they should engage an external party to conduct an evaluation. While it may seem strange for a management consultancy to say so, evaluations can, and often should, be conducted internally. Recent efforts to develop evaluation capability within government – such as the establishment of the Australian Centre for Evaluation at the federal level – are welcome developments.
Notwithstanding these trends, external parties are sometimes best placed to deliver evaluations, or at least to lead some monitoring and evaluation activities. In our view, external parties should be engaged when one or more of the following conditions apply (see Figure 2).
- Independence: When impartiality and objectivity are paramount, external evaluators can provide an independent and unbiased perspective that may be difficult for internal teams to achieve, especially in the context of large investment programs and sensitive or high-stakes evaluations. Perceived neutrality can enhance the credibility of evaluation findings and recommendations.
- Innovative approaches and fresh perspectives: Engaging external evaluators can help to bring new ideas and innovative approaches that internal teams might overlook, especially in the context of complex evaluations with many moving parts. A third party is often a useful source of insight on clever ways to develop a counterfactual or conduct robust attribution analysis.
- Technical expertise: External evaluators are often able to efficiently assemble teams with a wide range of skills – from sector-specific policy expertise to advanced statistical analysis – that are not readily available in-house. Diverse technical expertise can help to ensure rigorous design and delivery.
- Additional capacity: Sometimes organisations just need extra boots on the ground. When internal resources are stretched thin, external evaluators can provide the necessary bandwidth to carry out evaluation activities without overburdening existing staff. This can be done in different ways. An evaluation can be led internally with an external evaluator delivering discrete tasks. Or an evaluation can be led externally with client team-members included to bring close familiarity and subject matter expertise to the team. Blended or mixed teams can be a useful way to develop capability and assemble diverse teams.
- Context-specific credibility: When an evaluation needs to be grounded in community perspectives and owned by those with the most at stake, it can be helpful to engage evaluators that have a connection to, and credibility in, the local community. This is particularly important in evaluations involving First Nations peoples, where fostering trust, respecting cultural nuances, and ensuring that the evaluation approach aligns with community values and priorities can support self-determination and data sovereignty, and enhance the credibility and acceptance of evaluation findings.
Who should you outsource evaluations to?
Once the decision has been made to engage an external party to conduct an evaluation, the next step is to contact Nous… In all seriousness, though, choosing what kind of evaluator to engage is an important consideration.
The evaluation market can be broadly divided into four types of providers (see Table 1 below). Who to choose will of course depend on the specific facts of each case, but it is nonetheless helpful to be mindful of the typical strengths of each type of provider.
In some cases, it can be helpful to engage a partnership or consortium to access the strengths and mitigate the challenges associated with different types of providers. For example, Nous sometimes partners with universities to deliver evaluations, as noted in the case study below. By leveraging the specialist technical skills and knowledge of academics with the multi-disciplinary skills and resources available to us as a large consulting firm, we can deliver more insightful and higher quality evaluations.
| Evaluator | Typical strengths |
|---|---|
| Large consulting firms | Typically have resources and skilled teams to deliver large, complex or high-profile evaluations. Can assemble multidisciplinary expertise, including policy, evaluation, qualitative and quantitative research skills, and data analytics and visualisation capabilities. Bring a mix and balance of methodological rigour and pragmatism in recommendations. Typically have resources and capacity to deliver in tight timeframes and ensure continuity over long-term evaluations. |
| Boutique evaluation firms | Often have strong specialisation in evaluation methods and approaches. Can be more cost-effective than larger consulting firms. |
| Academics | Often leading experts in their specific discipline and fields (e.g., deep, theoretical and methodological expertise). Typically, a cost-effective option. |
| Sole traders | Often have valuable specialist expertise across narrow domains. Typically, a cost-effective option. |
CASE STUDY: Delivering evaluations in partnership with a university
Nous was engaged by the Australian Government Department of Health to evaluate two national psychosocial support programs for people living with severe mental illness. To assemble the team with the breadth of capabilities required to deliver this evaluation, Nous partnered with academics from the University of Sydney who had expertise in consumer engagement with people living with severe mental illness. This was critical to get rich qualitative data to assess the implementation, impact and cost effectiveness of the programs. The report has been made publicly available (see here).
How do you effectively commission evaluations?
Good practice in commissioning can help to ensure high-quality and fit-for-purpose evaluations. In our experience, the factors listed at Figure 3 are especially important in the commissioning process.
- Clear objectives and purpose – The objectives and purpose of the evaluation should form a clear and logical path to deciding when the evaluation should take place, the party best placed to conduct the evaluation, and the evaluation’s budget. Key questions to ask include: ‘where is the policy or program in the policy cycle?’ and ‘what will the evaluation results be used for?’
- Context and parameters understood – Prior to procurement, the scale, budget, timing and the evaluation type required must be understood. Providing this information can help evaluators design an appropriate method. It is important to strike a balance by providing sufficient information without being overly prescriptive. A prescriptive approach can deter evaluators from considering different ideas or techniques.
- Realistic scope expectations – Understanding the level of analysis and engagement needed from an evaluation is critical. Where complex quantitative analysis, large-scale consultations or ethics approval are required, this can involve a considerable cost. Often organisations have unrealistic expectations about what can be evaluated with the budget and data that is available. Working with an evaluation expert, whether internal or external, to assess the evaluability of the program can be a useful initial investment to determine whether a robust evaluation is possible.
- Collaboration during procurement process – Including collaboration between commissioning organisations and evaluators during the procurement and contract negotiation phases can create the space and conditions to develop and refine the evaluation design and realistic scope. Encouraging pre-tender engagement, such as through Expressions of Interest or Requests for Information, can foster collaborative understanding and a more tailored and fit-for-purpose RFQ and evaluation design.
- Evaluation expertise on the procurement panel – Compared with most projects undertaken by consultants, evaluations have a distinct set of methodological norms, epistemic standards and nomenclature. Including evaluation expertise on the proposal assessment panel can ensure that quality and methodological rigour are adequately assessed. Including people who will be responsible for translating evaluation recommendations into policy and program decisions is also valuable.
- Strategic allocation of evaluation resources – It is common practice to earmark a percentage of program funding to evaluations. While this is generally a good heuristic for evaluation effort, it can be more effective to invest more in strategically important initiatives and deprioritise smaller initiatives or those that have previously been evaluated.
- Consideration of long-term partnerships – Instead of opting for one-off evaluations, consider procuring enduring partners. Long-term collaborations, like Nous’ partnership with Emergency Recovery Victoria (see below), can provide continuity, deeper insight and capability development.
CASE STUDY: Nous’ evaluation partnership with Emergency Recovery Victoria
Since 2020, Nous has partnered with Emergency Recovery Victoria (ERV) to evaluate many recovery programs delivered in response to bushfire, storm, and flood events that have affected Victorian communities. Through this partnership we have brought our shared expertise to bear to scope, design and deliver evaluations that will give ERV and its partners the actionable insights they need. We have identified and mapped outcomes and lessons across a range of programs that sought to achieve a shared set of outcomes. The evaluations play an important role in assisting ERV and other recovery agencies to continuously improve disaster recovery efforts in Victoria and support an evaluation culture across the organisation.
Now the real work can begin...
Effective procurement is an important first step to deliver high-quality and fit-for-purpose evaluations that provide value for money and ensure that policies and programs are evidence-based. Good practice requires organisations to consider a series of strategic questions: when is outsourcing appropriate, what kind of provider is best placed to undertake the work, and how can they be commissioned effectively.
Get in touch to discuss how your organisation can more effectively commission evaluations.
Connect with Robert Sale on LinkedIn.
Prepared with input from Heidi Wilcoxon, Andrew Benoy, and Annette Madvig.
This is the first in our four-part series of articles on good practice evaluation. This series focuses on the steps you can take to ensure rigorous, high-quality evaluations. Download the full series here.
