In the session on transition to online, CDE Head Linda Amrane-Cooper and the Centre’s joint executive leads for research and dissemination, Stylianos Hatzipanagos and Alan Tait described London University’s marathon effort to put all its spring and summer 2020 assessments online. A team comprising Chris Cobb and Craig O’Callahan from University of London and Brunel University’s Mariann Rand Weaver gave a talk with the intriguing title ‘Assessment Re-booted’, and Martha Gibson from Heriot-Watt University in Edinburgh explained what she wished she had known about e-assessment when the crisis began. Finally, Simon Walker and Norbert Pachler from University College London addressed the question of assessment in these ‘times of disruption’.
Putting 110,000 Assessments Online
Professor Hatzipanagos began with a summary of some background research into online assessment, before and during the pandemic. Even 10 years ago, researchers were noting a growing student interest in replacing pen-and-paper exams with online equivalents, citing prompt feedback as a driver. However, student performance in online exams has until now been patchy, and some still have difficulty accessing the technology. Furthermore, some staff consider the COVID pandemic and subsequent shift online to have provided a ‘perfect storm’ for contract cheating. When the pandemic hit, the University of London faced the challenge of moving online more than 110,000 exams for 35,000 students in over 20 time zones at very short notice. The University therefore commissioned a project to explore students’ experience of online exams in 2020, with a focus on unseen assessments.
Dr Amrane-Cooper explained that the project is examining four aspects of assessment: student behaviour before and during the exams, including interactions with the VLE; students’, lecturers’ and examiners’ opinions, assessed through a student survey and interviews; outcomes; and some important operational issues. She then handed over to Professor Tait to discuss the survey. This was sent to about 30,000 students before results were issued and over 8,500 responses were received; most were from undergraduates, about half of whom attended one of the University’s Teaching Centres. Interestingly, student take-up of exams was slightly higher than it had been with traditional assessments in previous years, and most felt they had been able to demonstrate their learning. Many of those who chose not to take exams blamed the impact of the pandemic on their study in general rather than their assessment, and poor access to technology was rarely a problem. Furthermore, most students’ experience of online assessment was generally positive, and more than half the students expressed a preference for continuing to take exams online at home. Students with poor Internet access were, not surprisingly, significantly less likely to select this option. These results suggest that continuing online exams after the pandemic will be popular with students, which has important operational implications. Selecting the best formats for these exams; invigilation, proctoring and assessment offences; and improving infrastructure and communications will remain live issues for some time.
Chris Cobb set out a ‘2030 vision’ of assessment throughout the next decade. This report had been commissioned before the pandemic and it was clear that the shock move online had only speeded up a transition that was already underway. He suggested that assessment will need to stay relevant, adaptable and trustworthy as the context changes. Taking these in turn, one simple way in which e-assessment has established its relevance is in the switch from handwriting to keyboarding, and ‘digital’ also makes it easier to assess collaboration and group work digitally. E-assessments may overcome the disadvantage of some disabilities, and they can be adapted to student circumstances, examining students ‘anytime, anywhere’ whenever they are ready and taking prior learning into account.
One of 2020’s major challenges has been scaling up digital assessments to cover more students. Global student numbers are likely to increase dramatically in the next decade and it is not far-fetched to think of AI playing a significant role in their assessment by 2030. Staff and students need to be assured that digital assessments will, at the very least, not be more open to plagiarism and fraud than conventional ones, and there, trustworthy methods of validating student identity and data ownership will be key.
Dr Rand Weaver, Vice Provost of Education at Brunel, described e-assessment practices there starting when, in 2016, students were first allowed to bring their own devices to on-campus exams. In 2020, of course, all this had to be replaced with home-based exams. During the last four years the WISEflow assessment platform had been adopted for both coursework and exams, gradually until the pandemic and then completed almost instantaneously. All the over 800 exams held since lockdown were open book and time-limited, supported using an embedded chat that students found very valuable and used often. Around 80% of the students chose not to defer spring exams to August, but typically ‘disadvantaged’ groups – BAME, low-income and disabled ones – were most likely to do so. Staff were encouraged to include questions that required higher-order thinking, multiple-choice questions were minimised and randomised, and marking criteria adjusted to allow for open book conditions. Rand Weaver, like others, is now using 2020 as a ‘catalyst for re-imagining assessments’, maintaining a focus on the pedagogy of assessment with technology as the enabler, not the driver of change.
Dr O’Callahan’s experience at the University of London’s International Programmes has been very different. There, about 80% of some 35,000 students were assessed by examination only and they only had two months to achieve a transition that had taken Brunel four years. Furthermore, many students lived in countries with very low bandwidth, making a commercial platform like WISEflow impracticable. The only possible solution in the time available, was a Moodle-based one without proctoring, relying on Turnitin to pick up plagiarism. The technology worked well, although more irregularities were detected than in previous years, but Moodle was much less suited to handling the marking of very large numbers of assessments than a commercial platform would have been. This poses many questions for assessment in the ‘new normal’, where e-assessment will, at the very least, be a lot more common.
Transitioning to e-assessment: What I wished I had known at the outset
Ms Gibson’s presentation took delegates back to ‘the halcyon pre-COVID days’ in describing how, starting in 2015, the graduate business school at Heriot-Watt University had taken its assessment online. This works in a similar way to London’s International Programmes but on a smaller scale, with (in 2015) 12,000 registered students in 166 countries. A large majority of the students are mid-career professionals working flexibly towards MBAs, assessed by exams only with four exam sessions each year. The gradual transition to e-assessment arose in response to student feedback, to fit in with the students’ professional lives and business practices, improve exam security and reduce printing and shipping costs. There were, of course, challenges, not least the enormous variation across the world of students’ context and experience and assumptions about what ‘e-assessment’ would actually mean.
Rather than taking the opportunity to re-design the whole assessment strategy, they chose to deliver existing face-to-face exams through an e-assessment platform. This led to technical issues; whole courses, including the core Economics module, were initially excluded from the system because the platform did not allow students to submit diagrams. There was also no way to mitigate technical problems during the exams. Initial student feedback was distinctly patchy, but once staff and students had experienced the system few wanted to return to pen and paper, and uptake of e-exams increased gradually from one exam session to the next.
So, after a generally successful launch, what did Ms Gibson wish she had known before she started the project? In brief:
- What all stakeholders understood by the term ‘e-assessment’
- The situation and the subsequent challenges in each country and at each local exam centre
- How other UK-based global programmes were managing the same transition
- How much practice, preparation and training students and staff would need
And what would she do if she was starting again from scratch? She would use a hybrid model of remote invigilation and test centres to reach a wider group of students, make e-assessment the default and, crucially, take the opportunity of a ‘root and branch’ refresh of assessment design rather than re-purposing existing paper-based exams. She recommended delegates to join the e-Assessment Association, which is free.
Dynamic responses to assessment in times of disruption
The final presentation, in contrast, focused on how the pandemic had disrupted assessment at UCL, and what can be learned from this for future years. UCL is a global university, and very many of its overseas students have been unable or unwilling to return to the UK. In this context, the dominance of exams as a mode of assessment is, as Professor Pachler explained, ‘perhaps unhelpful’, and assessment is likely to place too many demands on students. E-assessment is clearly the way forward, but the crisis has merely accelerated this change in strategy. It is clearly led by pedagogy, with technology as an enabler.
In a normal year, students at UCL would take about 2500 exams between March and June. In 2020 these were replaced by a mixture of coursework and timed online assessments, with a discipline-specific ‘capstone assessment’ for all first years. This change to the previous exam-based system was a key part of a broader University-wide strategy for crisis management.
Looking beyond the immediate crisis, UCL is now setting out its strategy for the academic year that has just started in an environment that is still disrupted and subject to rapid and unpredictable change. Professor Walker described how their online assessment model was developing ‘at scale and at pace’ in a community the size of a small town. Assessment load for staff and students is a serious issue. Planning for 2020-21 started at the beginning of the spring lockdown by developing a guide for alternative assessment strategies beyond exams. A ‘connected learning essentials’ course was developed for staff training, together with a model for calculating assessment load to ensure equivalence across different programmes.
This is now used when new modules are developed. An evaluation of staff and student experience of online assessment generated positive feedback but also highlighted a few hassles; many students reported being more stressed by these exams than the paper-based equivalents.
The final piece of UCL’s online assessment strategy for 2020-21 and beyond will be a dedicated digital assessment platform. This supports the whole assessment cycle from design through to moderation and can be used in a wide range of languages and disciplines. Following proof of concept and piloting the system should be available for all students taking exams at the end of the academic year. In a large institution like UCL, maintaining a common assessment strategy, improving communication with staff and students, and ensuring prompt feedback are key, and the platform should support all these.