When Dundee and Angus College was brought to a halt earlier this year by a ransomware attack, the leadership team quickly realised they were tackling a situation that wasn’t in any handbook.
With no means of digital communication with staff or students, they had to think on their feet and get back to basics.
“Nothing had prepared us for the crisis we headed into,”
admits principal, Simon Hewitt, who believes that the strong college culture helped speed the recovery process.
“It's a big organisation, with 1,000 staff, and I've never seen them pull together as much. With great teamwork, we rebuilt the digital element of our college in just less than five days. It was incredible,”
says Hewitt, who, at the time, was vice-principal in charge of ICT and responsible for the technical restoration.
It was far from a simple process, however.
Back to basics
“The attack happened on Friday, 31 January and we had to send everyone home. I remember that at 02:20 on Saturday morning it dawned on me that, in a digital sense, there was no college; everything had been wiped. That was a pretty low moment,”
reflects Hewitt.
Ironically, number one on the business continuity plan was to set up an email conversation for key staff. But nothing digital worked except the website, which was hosted externally, so social media and the website provided the only means of contacting students.
To keep staff informed, Hewitt and the business continuity lead, Jaki Carneigie, created a WhatsApp chat to communicate with the business continuity team. Then, once email communications were restored, they were quickly inundated by offers of help from staff and, as word spread locally, from the wider community, too.
“I couldn’t believe the level of support and offers of help we received from all over the sector and beyond,”
says Hewitt.
“At the beginning we said ‘thanks, but we’re fine’. Then, on day two we realised we needed to make use of more people, so handling those enquires became a priority. For example, the University of Dundee offered to send a team and helped us rebuild a lot of the laptops.
Open and honest communications
“By Sunday afternoon, we had a list of priorities with communications at the top,”
says Hewitt.
“And even if we had nothing new to say, we continued communicating to the staff via a new temporary portal and to the students, local media and the wider public via our website and social media.”
This open communications strategy seemed to have helped preserve the college’s reputation and has since been recognised as best practice, with praise from a variety of sources.
Hewitt explains:
“There were people who wanted to keep the cyber attack under wraps and limit what we said about it, but the team was clear that we needed to be honest and up-front.
The feedback we've since had from Scottish Government, from the National Cyber Security Centre and colleagues has been incredible; they're astonished by how well we responded and that we recovered so quickly.”
Rebuilding better
So, with a small army of people working initially off a to-do list of Post-it notes, the rebuild process began over the weekend, but not on a like-for-like basis. Hewitt and the team realised that the cyber attack, devastating through it was, provided a unique opportunity.
“There is a digital strategy in place, so we knew the direction of travel over the next two years. We managed to convince the board to shift the budget forward to implement those planned changes immediately. It helped that we had asked the Jisc cyber security team to provide data analysis of the attack and their recommendations backed up what we were saying. Their support throughout was outstanding.
“So, we started to look at rebuilding the systems in a way we envisaged they might look in the future - to make us more resilient and to enable more remote access for staff and students. We rolled out OneDrive and Teams right across the organisation, and we moved to a cloud-first approach so that, if this ever happens again, we won’t lose all our files
“Of course, all that had a financial impact and we had to use money that had been earmarked for other projects, but it was the right thing to do. Little did we know a pandemic was coming, so it turned out that, from that point of view, the cyber attack was a blessing in disguise.”
A bit of luck, but no rest
Fortunately, Hewitt and his team discovered that the malware which paralysed the college hadn’t caused as much damage as originally thought. He says:
“It didn’t seem to affect anything that was still live and transferring data, so while we lost all the file servers, and the front end of the VLE, we found the database behind the VLE was intact and so was the student records database.
“It was relief to discover things were not quite as bad as they could have been, but there was still plenty of stress and very little sleep.”
After working through the weekend and telling students via social media and the college website that Monday classes were cancelled, all staff were called in for a meeting. Filtered through a single entrance so nobody was missed, they were directed into one of the large lecture theatres.
Hewitt remembers feeling very emotional before he addressed them all that morning.
“I had to get up in front of hundreds of staff and tell them what happened, to tell them that all their files, everything, had gone. I'll never forget the gasp - the intake of breath. That was tough.”
To pay or not to pay
Aside from dealing with the practical and logistical fall-out, Hewitt spent a lot of time talking to police and dealing with a ransom demand for the return of the lost files.
“The cyber attackers had managed to get access to our bank account and knew how much money we had in it, which was the budget for the whole year. They demanded a ransom of exactly that amount, which we were never going to be able to pay.”
Heartwarming moments
But there were relative ‘highs’ too:
“On the Thursday, when things were starting to come back online, myself and a couple other members of the senior team went round every work room and handed out chocolate and sweets, just to say, ‘thank you’.
“Staff were thanking us back, giving us hugs and empathising with what we had gone through and the stress we’d been under. That's not normal! But it was, again, a wonderful testament to the college culture.”
Lessons learned
Part of that ethos, says Hewitt, is that the college is a “learning environment” with no “blame” culture. The cyber attack certainly offered a few lessons, not least to the IT team.
“Before our attack, we thought we were pretty robust about cyber security,”
says Hewitt.
“At the end of 2019 we were proud of the fact we had got Cyber Essentials in place, but it didn’t ‘save’ us. We’ve got Cyber Essentials Plus now, but I think it’s easy to get caught up in certifications and to become complacent.
“I know that mistakes by end users present one of the biggest security risks, and we had put staff through a fake phishing attempt to help with that, but our ransomware attack wasn’t down to any user - it happened because one of the servers wasn’t patched; a simple mistake that caused total chaos.
“We didn't try to hide and shirk – we faced it head on and knew there would be lessons to learn, and part of that was an internal audit. We are implementing recommendations from that now and one of the last pieces in the jigsaw is to recruit for a dedicated cyber security role.”
Dealing with emotions
Having had time to reflect, it was the emotional effect on people - the unpredictable way in which people react to pressure – that sticks in Hewitt’s mind.
“No amount of training or documentation prepared us for how people reacted,”
he concludes.
“People with usually cool heads panicked, while others unexpectedly emerged with skills we needed. It was a proper emotional rollercoaster.”
Yet, now, nine months on, Hewitt speaks about that time humbly, but with great pride - pride in his staff and in the teamworking culture at the college which meant that, despite the intense pressure, everyone pulled together.
He is certain that the sense of community at the college was in large part the reason Dundee and Angus was able to recover so quickly. “There was definitely a feeling of ‘we're all in this together’ which really shows the value of the college culture.
“I'm a heart-on-the-sleeve kind of guy and this experience has been brutal; it’s probably been the most challenging time of my career, but I think that, like everyone else at the college, it has underlined what's important and that’s doing the right thing for the students, no matter what.”
To hear more from Simon Hewitt, sign up for Jisc’s security conference, free to attend online from 3-5 November 2020. Hewitt is speaking at 10:15 on 4 November 2020.
No comments:
Post a Comment