Belfast, United Kingdom
Improving student satisfaction and retention is a long-standing purpose at Ulster University, Northern Ireland’s civic university, with 27,000 students distributed across four locations in the region, within partnership campuses in London and Birmingham, and enrolled in fully online programs.
Ulster is the first institution in the European Union to use Blackboard Predict, a solution that leverages data and advanced analytics to identify at-risk students. The university has implemented a pilot project on predictive analytics in record time – less than a year. Through the application of a predictive model, this initiative allowed professors, instructors and student advisors to make early interventions when noticing the first signs of a struggle.
Student Retention, an Ongoing Effort
At Ulster, student retention is not a new concern, but they have been adopting a more structured approach in recent years. The institution has historically been involved in a UK-wide higher education project published in 2013 called What works? Supported by the Paul Hamlyn Foundation, the initiative examined how higher education providers can improve student retention and success.
Ulster had a number of case studies aligned to that work, and some of the strategies that emerged from these were building engagement through partnerships between staff and students, promoting peer support and creating a sense of belonging, among others.
As a result, the university has taken a strategic, long-term, longitudinal approach to student retention, which from now on is going to be supplemented with real-time data-based decision making and interventions. This will be done through the application of predictive analytics, using a predictive model based on past student data to identify learners who are likely to have difficulties and drop out in the future.
“What we are doing is enhancing our approach with real-time, just-in-time data. We are very respectful that several areas of our institution have very well-designed intervention and retention strategies. However, we are seeing parts of the university that have much higher drop-out rates than we would like to see,” says Andrew Jaffrey, head of the Office for Digital Learning at Ulster University. “What we are trying to do with Blackboard Predict is to provide a standardized platform and to encourage data-informed decision making and more timely intervention strategies.”
This is not the first time that Ulster has used learning analytics to improve the learning experience. A previous work with descriptive and diagnostic analytics has allowed them to investigate the “what happened” and “why it happened,” as Jaffrey explains. “We have looked at descriptive and diagnostic analytics to tell us how our learning management system was being used and how the different tools and technologies that support learning were being used,” he notes.
Until the day they were challenged by a new pro-vice-chancellor for education to think about adopting a more student-focused predictive approach. A project board was then established, with wide representation from faculties across the university, but also from relevant internal stakeholders and professional services departments such as the student support department, IT services, the student administration and the quality management unit.
“One of the challenges with learning analytics projects is about project ownership and where in the university that project should be established. By bringing all those stakeholders into the project, we were able to have really valuable discussions,” Jaffrey points out. From the moment they decided predictive analytics was the way to go, the project timeline developed rapidly. Ulster signed a pilot contract with Blackboard in summer 2017, with the expectation of going live within the first semester of 2018 academic year.
“A real motivation for the project was to better understand our data and to make sure that our assumptions about our data were accurate. So, a large part of the early work was around data and data cleansing, making sure that the data we were sending to the predictive model was accurate. That took a lot longer than we expected. But once we had got the data cleansed and in a good format, the predictive model was generated very quickly and we had it available and tested early in January of 2018,” explains Jaffrey.
From this moment on, they started the roll-out process and began raising awareness across the institution. The period from January to April was a time to discuss data ethics, governance and policy, as well as to promote adequate training for the professionals that were going to support the initiative. Only then the project was made institutionally available.
Anticipate student drop out with predictive analytics. Learn how.
Challenges and Issues to Look Out For
One of the biggest challenges during implementation of the project, according to Jaffrey, was system integration. “We are working with Blackboard Predict, which is hosted within Amazon Web Services, and we had to set up specific system integrations that ensured that our data was going to be protected and in line with European Union legislation and law. That system integration took a long time to set up technically. We had to establish secure transfer mechanisms to support the transfer of data between our data centers and Amazon Web Services,” Jaffrey notes.
In contrast, the agility in the developing and testing of the predictive model was a positive aspect of the project. “Because the Predictive model was based on 80% of the last four years of student data, 20% was held back for testing. We were able to run the model through that 20% of known outcomes and the model passed the test much more quickly than we expected.”
That meant that conversations accelerated quickly within the project team, because all of the sudden they had dashboards and a live model to demonstrate to stakeholders. As a result, discussions on ethics and guidelines for learning analytics policy development were reopened.
“Suddenly, dashboards became available, and that enhances conversations. At that point, we had a lot of engagement with our student union and student support around the ethical use of student data,” Jaffrey notes. “We put a lot of thought into how dashboards would be perceived by students and the potential issues of students being impacted negatively if they were seeing that they were performing poorly. For example, we’ve had several discussions around the language that we would use in student support interventions to make sure that we are as inclusive and supportive as possible.”
The extent of the work that needed to be completed after the dashboards became available in the university’s controlled environment was something Jaffrey did not expect at the beginning of the project. The institution wanted to be reassured that the projections they were seeing were accurate and that they have had enough time to discuss and review them before making them available to students.
“Quite often, the data shows us a narrow view of what’s going on in a student’s life. I think that it’s really important to consider the human aspect. We are dealing with human beings, not a number, and we need to be respectful of that,” Jaffrey notes.
Anticipating the Future
At Ulster, the instructors within Blackboard Learn and the teaching staff who have responsibility for modules in the digital learning environment have full access to the analytics dashboard. “They are very much involved with interventions and monitoring how their students are engaging with the material,” Jaffrey notes.
He explains that some areas of the institution have very well-defined student advisors, while others do not. “The decision we made was to give everybody who is teaching, anybody who is supporting a student, equal access to the predictive solution. We didn’t want to just have the solution locked down to those with a student advisory role. So, academic staff have access to the dashboards, but so does any member of staff that is supporting a student throughout their journey at Ulster.”
When it comes to how to intervene when an at-risk student is identified, there’s no one-size-fits-all solution at the university. “We see Blackboard Predict very much as something that starts a conversation with a student. So academic colleagues are intervening in many different ways across the institution. We’re seeing the data modeling aspects and the predictive analytics as a way of starting that conversation and having that human contact,” Jaffrey explains.
That is a key point to understanding Ulster’s vision on learning analytics: at the university, data is not seen as a definitive answer, but as a starting point. They want to learn more about data and have more conversations about it. Some of these conversations, Jaffrey says, will certainly be challenging, but that’s a good thing.
“People have many different views and we think about a lot that’s happening in the media now, with Cambridge Analytics and Facebook, that really heightens people’s conversation around data ethics and privacy. And learning analytics sits on top of deeper layers of assumptions and beliefs that academic colleagues have,” Jaffrey explains. However, he thinks that’s a rich place to have conversations and to be respectful of people’s different views.
“With any database project, we should recognize that data is imperfect and we must treat that data with humility, because it is only giving us a narrow picture and a narrow view of what’s going on with a human being,” says Jaffrey. “And there’s lots of external factors going on with a student, well beyond anything that we can hope to measure within an institution. I think we all, as a sector, need to be respectful of that and treat data respectfully when we’re engaging with these sorts of projects.”
Andrew Jaffrey, Head of the Office for Digital Learning at Ulster University
AFP Paul Faith
1 EU GDPR Portal. (n.d.). Key Changes with the General Data Protection Regulation. Retrieved April 28, 2018, from https://www.eugdpr.org/key-changes.html.