Research and writing in the shadow of reporting and evaluation frameworks

With ERA 2018 just around the corner I’ve been spending a bit of time with various researchers within the Faculty, discussing strands of ERA strategy in an attempt to define and focus on ‘excellence,’ and more of it going forward. Discussion inevitably touched upon a few core points of tension that result from the pressures of institutional reporting expectations competing with individual goals and career stages.

How, for example, to reconcile the pragmatics of publishing and the publishing industry with the emphasis on publishing for excellence and impact?

How to navigate the quantity v’s quality dichotomy, when both are required in different contexts and actively sought in different circumstances?

What might I need to sacrifice in order to achieve excellence?

When and how are excellence and impact competing against each other?

Difficult, relevant questions that have been raised by different members of the research community I work with on a daily basis.

Today’s post will share an excellent article by UK-based researcher, Sharon McCulloch, which touches on a number of these questions, but focuses on the all-important process on which a significant part of impact and excellence depends upon – academic writing itself. Her piece, “The importance of being REF-able: academic writing under pressure from a culture of counting” draws on a two year study that examined academics’ writing practices in the contemporary workplace.

It details “tensions around the ways in which managerial practices interact with academics’ individual career goals, disciplinary values and sense of scholarly identity,” which I’m sure regular readers of this blog could well identify with.

The drive toward research impact can be evaluated from many angles and a thorough consideration of the concept must include the experience of researchers themselves as writers and communicators.

The following article was first published in the always excellent LSE Impact Blog on February 9th, 2017 and is shared here under a Creative Commons Attribution-ShareAlike 3.0 Unported License.


Academic life is diverse, including research, scholarship, teaching, and public engagement. But the principal role of an academic is to produce, shape and distribute knowledge. Writing is central to this endeavor, but academics’ writing practices have come under pressure from several directions, such as the increasing marketisation of higher education and changes in the digital landscape, both of which have brought about new forms of writing as well as changes to existing practices.

For the past two years I have been working on an ESRC-funded project investigating academics’ writing practices in the contemporary university workplace. The project looks at how knowledge is produced through academics’ writing practices, and how these are shaped by, among other things, managerial practices and evaluation frameworks. We have interviewed 51 academics in three disciplines (mathematics, history and marketing) across three higher education institutions in England, as well as administrative staff and heads of departments. These interviews have revealed tensions around the ways in which managerial practices interact with academics’ individual career goals, disciplinary values and sense of scholarly identity.

Universities in the UK are subject to a national Research Excellence Framework (REF) aimed at rating the quality of research and allocating funding accordingly, with higher rated institutions receiving more funding. As well as direct funding, a high score on the REF also links to rankings and league tables, which in turn affect an institution’s ability to raise income from tuition fees. Given the importance of high REF scores, most universities and departments have policies in place to encourage their academic staff to produce work likely to score highly in the REF.

Our study found that academics’ capacity for career advancement was closely coupled to their universities’ strategic interests in performing well on the REF. For example, during probationary periods, new lecturers were required to publish certain numbers of papers of a specified quality. For academics working in marketing, quality was determined using the Chartered Association of Business Schools’ (ABS) annually published Academic Journal Guide, which ranks business and management journals on a star-rating system similar to that employed by the REF. This target list of journals was used in all three of the marketing departments participating in the study, and the star-rating system employed by the ABS was deeply embedded in discourse of the marketing academics we spoke to about scholarly writing and academic success. Each talked about their own publications in these terms, as seen in the comment below:

“I don’t get any hours for writing. I don’t get any hours for research whatsoever. So basically, unless your work is at least three-star, four-star, then you don’t get any hours for it because although it’s two-star material and it is REF-able, they’re only interested in three and four-star.” (Lecturer in Marketing)

This comment illustrates both the extent to which the REF has become naturalised in the rhetoric of what academic success is understood to mean, and the dangers of any ranking system. Our participants repeatedly used the adjective “REF-able”, meaning that one has enough publications of sufficient quality within the REF period (five to six years) to be included in the department’s submission to the REF, as shorthand for talking about their career status. Although being REF-able was seen as a prized benchmark for academic success, it was far from being sufficient. As the comment above shows, having enough publications (up to four in the 2014 exercise) to be REF-able is unhelpful if they happen to be rated below three-star level, otherwise defined as “internationally excellent”.

It also became clear that, although heads of department described their departmental systems of evaluating academics’ publications as ways of “rewarding” good publications, most academics saw this performance management as something closer to a threat than a reward. They talked of struggling to achieve what they saw as a small and moving target. For example, some key target journals in marketing moved down the ABS rankings in 2015, yet the pressure on academics to “hit a four-star” remained. Even academics who were, by these measures, performing well expressed anxiety about being unable to sustain such a high level of ‘excellence’ year in, year out, and about what might happen if they failed to do so.

Evidence emerged from our interviews of what could be described as strategic behaviour in regard to meeting these performance targets around scholarly publication, but while strategic behaviour enabled academics to meet their targets, they paid a high price for this in terms of their sense of disciplinary identity. One professor in marketing described publishing her research in journals outside of her disciplinary area: “now I target management journals, which is one way of hitting a four star”. This enabled her to maximize the prospects of career advancement, but it gave her something of an identity crisis about who she was as a scholar.

Marketing was not the only discipline where fulfilling managerial demands pertaining to research evaluation conflicted with disciplinary values. Over and over again, academics described peer-reviewed journal articles when asked about the sort of writing they were expected to produce. However, what they were expected to produce and what they wanted to produce was not always the same. Historians talked about the scholarly monograph as their most valued form of disciplinary writing. One historian described the monograph as “the heavyweight, solely authored piece of research work, which is usually the result of years of research in archives”. One problem is that the time it takes to produce a monograph may extend beyond the REF period, so in order to be REF-able, historians also need to be publishing articles that can be written relatively quickly allowing more to be produced in the same timeframe.

Another historian, in her first lecturing post, described a tension relating to this for her first published book, based on her PhD research. She saw this first monograph as potentially career-defining, so had wanted to do it well, to augment and extend her doctoral research rather than merely making minimal changes and publishing it in book form. However, she also knew that without the book, she would struggle to secure permanent employment, so felt pressured to compromise the quality of the research in order to publish quickly and make progress in her career.

We also asked the academics in our study about their use of digital technologies, including whether they engaged in any forms of online writing by, for example, contributing to blogs, tweeting or other emerging means of digital scholarship. Some refrained from these new forms of writing on the grounds that they were perceived to be trivial or self-aggrandizing. Others expressed interest and enthusiasm but nevertheless did not devote much, if any, time to these digital platforms. Their comments about lack of time were often qualified by reference to the belief that such writing did not count or was not valued, as seen in the comment below:

“A lot of the work is grey literature where people have written blog pieces. I think that’s opened my eyes to what’s possible in that area but yes, if there’s time – I think it’s always a question of time. Again, that work is not valued by the university as far as I can see.” (Professor in Mathematics)

Non-traditional genres of academic writing were not perceived to meet the criteria departments have in mind when they stipulate that, for career progression, an academic needs a track record of ‘good publications’. Understanding of what counts as writing worth doing does not stretch to emerging online genres, despite the increased attention paid by universities to public engagement and dissemination of research findings to a wider, non-academic audience.

The picture that emerges is one in which academics are positioned as managed professionals whose personal goals are expected to be closely aligned with the university’s objectives to perform well in the REF, move up the league tables, attract students and secure income. In a neoliberal culture of measuring outputs, the range of forms of knowledge creation that are valued appears to be narrowing. High-prestige journal articles are seen as essential to career success, and should be ranked at three or four-star to secure rewards such as promotion or time for writing. The academics in this study strived to shape their writing around these targets, even though they saw them as unrealistic or out of sync with disciplinary values. Because scholarly writing and disciplinary identity are so closely intertwined, for many academics, this pressure engendered something of an existential crisis about the true purpose of their writing.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s