By Diane Levine
In their 2020 chapter “The South against the destroying machine”, Lara Hofner takes an interdisciplinary approach to reflecting on the social realities of the Minority World, the ways in which they are hegemonic and violent, and the contrasting social realities of the Majority World, considered ‘oppressed’ (see Hofner in Baumann & Bultmann, 2020). In this blog post I reflect on the challenges of evaluating the MNS Disorders in Guyana’s Jails project as we saw them at the outset, then share some of the key messages emerging from the mid-point evaluation, and consider some of the challenges we will face in the remainder of the project in ensuring evaluation does not become part of the “destroying machine”. [Note: I do not sit directly within the research team, which I hope has given me some small sense of distance and objectivity in delivering evaluative activity.]
What was
At the project launch stage the team’s planned evaluation and impact activities were founded on some shared key principles:
- We collaborate and align our efforts for the benefit of the project as a whole wherever possible,
- We ensure equity of access to data, including by considering gender, socio-economic and socio-cultural dimensions to our findings.
- Our research emerges from meaningful understandings of the complex environments in which we operate.
- We promote decolonising methods and perspectives.
- We learn continuously by analysing and reflecting on the specific and changing circumstances in which we operate.
- We harmonize with our colleagues outside the academy, committing to co-ordinating our efforts with others in the same space on their advice.
The challenge ahead was not underestimated by team members. Impact and evaluation have already been problematized widely in the academy (e.g. Aguinis et al, 2014). With particular reference to this project were conceptualisations of impact that rely wholly on ideas of rationality and control that provide an unfortunately fantastical security in a context that does not in reality allow for non-linear ‘contradiction, complexity, or paradox’ (e.g. Shahjahan & Wagner, 2018, p.g.3). We all saw that this formulation made an incontrovertible link between the rational and the conqueror, and brought us dangerously into colonial practice: in this framing everything must be manageable, observable, knowable, and measurable, as the team sought to identify causal linkages between intersectional complexities and ‘impactful’ intervention.
As was expected across the funding landscape at the time (2018/19), the team intended to produce an evolutionary logical framework, and emergent classical Theory of Change goals that would: i) model pathways to impact, explaining the potential connection between activity, output, outcome, and impact, ii) provide rationales on how implemented activities and inputs are likely to lead to our desired outcomes, and iii) make assumptions and constraints explicit. The original logframe looked something like this:
Attempts to conceptualise impact in the context of a decolonial imperative have aimed to demonstrate multi-stratified perspectives of reality (e.g. Izutsu, 2008), and alternative ways of knowing that we cannot normally see through the common impact lenses of, say, policy citation (see Śūnyatā, as explicated in Shahjahan & Wagner, 2018). Change that might arise from our research activities as viewed through Śūnyatā’s lens is not change in itself. Rather, change depend for existence on everything else.
We realised that we would need to learn the lessons being taught to us by the limits and failures of tools such as logical frameworks and Theories of Change, whilst acknowledging our commitments to our funders, colleagues’ careers, institutional progression, and our partners. Following their first fieldtrip (March 2019) the team began to ask themselves some difficult questions:
- Can we understand and evaluate in a pluriversal way that surfaces the interconnectedness between us all in the Guyanese context?
- Can we accept the discomfort that our work may not ‘make a difference’ in the ways we conceptualise ‘difference-making’?
Alongside finding ways of addressing these questions, the team realised that parallel systems might need to be run for capturing the pragmatic requirements of funders and institutions. They wanted to formulate these new ways of evaluating and knowing without sacrificing their integrity. This seemed to me an excellent position from which to begin my observation (and learning) journey.
What is now
One of the parallel systems to which we had committed was a mid-point evaluation. Our original conceptualisation was of something that would be delivered in Guyana, with Guyanese stakeholders. Sadly, in March 2020 we had to rapidly re-formulate our approach with the onset of Covid-19. Fieldwork, workshops, focus groups and consultation would no longer be possible in the way we had envisaged. Not only that, but uncertainty within the Guyanese socio-political context, and associated significant workloads for everyone, meant that we could not in fairness ask people to give up their time for long virtual workshops.
In the interests of pragmatism we opted for a light touch mid-point evaluation comprising a content analysis of all meeting minutes to summer 2020, 1:1 interviews of all team members willing to speak, and a summary that would then be reflected to our Advisory Board for comment, critique, and critical friendship. Six key themes emerged.
What will be?
So what about next steps? Well, there are some practical things we need to do. For example, for large scale projects, we need to begin to consider costing/building in professional support for those gathering data in the field (including in archives), or possible training modules available to all teams in managing emotional responses to this kind of high-stakes work.
But the significant, intersectional task ahead for evaluation will be to continue to recognise that “evaluation is unavoidably and simultaneously in dialog with the prevailing contexts of colonization and decolonization vis-à-vis the location and moment in which it occurs” (Marama Cavino, 2013). We need to build a culturally-meaningful, Caribbean-aligned, model of evaluation that meets Guyanese needs, as well as our original commitments. Watch this space!
Dr Diane Levine is the Deputy Director and Manager of the Leicester Institute for Advanced Studies at the University of Leicester.