Dr. Anandha Gopalan, Principal Teaching Fellow and Dr. Robert Chatley, Deputy DUGS, Department of Computing

Before you plan an online exam, you should consult your Faculty EdTech team for information on the system for remote exams your faculty has put in place.

Setting-up the remote exams

The Department of Computing were one of the first departments to go online with their exams, so we had to think on our feet a little. We have exams at the end of term for our MSc, 3rd and 4th year students. We had about 48 hours notice with which to convert our face-to-face exams to online versions and run them, as scheduled from 16^th to 20^th March to March 20th. In order to do this, it was decided to run them as open-book, timed assessments.  These were set up in CATE (our departmental VLE) as 'timed courseworks'.  In terms of submission, students would treat them as a coursework on our system; when exam time began they were able to download the exam paper from the VLE, work on it, then scan it back into the system and submit it on the VLE as a single PDF document.

Familiarising and supporting students

In order to make students more comfortable with the arrangement we shifted the exam scheduled for the morning of 16^th March later so that the morning could be used for a
'mock exam'. This was primarily to allow students to familiarise themselves with following the exam download instructions writing on a few sheets of paper/tablet to mimic an actual exam and then scanning and uploading it to the VLE. This turned out to be very important so that the students could work out a good process for scanning, assembling and uploading their work.

We also created a dedicated Piazza board and enrolled all students who would be taking these exams. Detailed instructions were uploaded to students on these boards. The instructions for scanning the final pdf were also provided and quite a few of these scanning tools were provided by the students . The Piazza boards were also a great place for students to ask any questions and for other students to see the responses. We made it clear to the students that the course-wide Piazza boards themselves were off-limits during exams and that they did not have an option to ask examiners any questions. If they did spot an error they could email the examiners (the exam timetable was updated with this information) and then examiners would put a post on Piazza stating 'EXAM MONITORING' or 'EXAM ERRATA' where this would be clarified. General enquiries or questions were discouraged as we did not want the examiner to be answering lots of individual questions from candidates during the exam, especially when broadcasting a message to the entire class is difficult.

In order to address any technical issues during the exams, a rota of Teaching Fellows monitored a shared email list (doc-tf-list@imperial.ac.uk) and one of the Teaching Fellows on duty would respond as necessary and cc the Teaching Fellows. The key issues we found in general were: files too large, faint scanning, scanner crashing, slow upload speeds, internet failure etc. In advance, as well as when they asked for help, we advised students to try and gather any evidence as necessary and submit them to us, after which we would forward this as necessary to the respective Senior Tutors. Issues could then  be discussed in the mitigating circumstances panel and pre-exam board meeting on a case by case basis. We also provided template 'cover sheets' (one of which was created by students).

As and when we learnt new things, such as regarding gathering of evidence, etc. we updated the instructions on the Piazza board and advised students by clicking the ‘e-mail notification immediately’ button.  When communicating beware of how the message is received by students, as rep fed back “Piazza is very aggressive on email notification whatever setting you set, and piazza in exam period is worse. So, if new posts are seen, those updates often took a long time to get shared.”

Providing access to examiners

The workflow used to provide examiners with access to the scripts after the exams was the following:

• Exams were downloaded from the VLE and then processed to change the submission from timed_cw.pdf to <CID>.pdf
• This pdf was then uploaded to Box (software for which we have licence from the College) into a folder for that particular class in that particular exam (for exam, for a student on the Computing 3rd year BEng in course C347 it was uploaded into folder b3 (for BEng 3rd year) of course‘e347
• Access to the above exam folder was given to the respective examiners, who agree their workflow as first and second markers.
• The marked scripts are left on Box and the marks uploaded as usual to our exams system.

Lessons learnt

The week of exams generally went well. Some of the things we learnt and improved include:

• Partnership with students, and specifically student reps, at such times is very valuable as they can play an integral role in informing and communicating decisions as well as supporting peers.

"I am convinced that without the reps the exams would not have been able to take place because they were crucial on the communication front (answering a lot of questions in the background, gathering quick and early feedback on points being worked on) and shaped the process in a way that would fit the student body (e.g. scanning policy)... they were also very useful listening to student concerns, sometimes just being there to listen and calm students was what was really needed, and sometimes there were real issues that had to be funnelled and turned into a coherent speech back to staff. I simply don’t think students would have been able to take the exams without them, both logistically and mentally"

• The impact on students’ emotional well-being of quickly transitioning to unfamiliar assessment formats and conditions in uncertain circumstances should not be underestimated. 

"Students were and are still anxious today of the multiple effects running exams remotely had. Some were doing with real issues, like choosing between moving to a better quarantine place or take exams and struggled to find support in those times on those “less-practical” issues. Practical questions were handled really well on Piazza (visible and open to all, allowing edits by the Teaching Fellows in a visible and open way)."

• Providing on-going, responsive student support via familiar communication channels is paramount for reassuring students and reducing anxiety. 

"Students were also absolutely exhausted halfway through the week with the situation and the Piazza post had a considerable impact in boosting the spirit, a lot more than staff is suspecting I think… The same can be said about how useful the very humane involvement of the Teaching Fellows (on Piazza notably). It wasn’t the traditional “exams are stressful [ it’s ok]” messages or replies, but everyone was listening through that, to what the underlying issues were and obviously trying to fix them and collaborate."

•  A specific source of anxiety and questions has been local interpretation of the safety net policy.
  
• Some students did post to the course Piazza boards during the exam and we had to shut it down and remind them not to do so.
• Students submitting late sometimes forgot about capturing any evidence, as necessary.
• Students had to declare on the VLE that the work was their own. For the summer exams we asked all students to sign an honour code which almost all of them have done.
• College guidance on remote exams came later, so we had to update our instructions to mention that students could not use any electronic aid (e.g. tablets or laptops) for their exams, unless they were authorised to do so.  We will include even more 'scanning time' to align with new College guidance.   
• We provided students with multiple attempts at doing a practice test for the summer exams (3 in total).
• The guidance provided was tightened up and provided not just as one big Piazza post but in multiple posts for easier readability.
• It is still very important to keep up good communication and to reassure students that we are taking this assessment approach to ensure overall fairness and quality of their education (especially since many students were worried that exams may have become harder). 
• Most of the above relates to the exams done using our VLE. We have another system called Answerbook, predominantly for ‘code-based’ electronic exams.  This doesn’t require scanning time as students submit on the system directly, but is currently difficult for exams involving handwritten maths or diagrams, as input is typed.

Key points

The main things that we tried to focus on were:

a) making the experience as smooth and familiar as possible for the students (just like regular coursework submission, submitting a single document) 

b) establishing a procedure to deal with problems as best we could as they came up (not trying to put in place policies for all possible eventualities up-front) 

c) working together with the students to make things work, and taking on their suggestions where they would benefit others.

"It’s not just the communication between students that improved in this period, it was also the student rep participation in work that had to be done which made sure the policies were fitter for the students and more quickly integrated by having a tight feedback loop."