We use nine measures of performance, covering all stages of the student life cycle, to put together a league table for 54 subjects. We regard each provider of a subject as a department and ask each provider to tell us which of their students count within each department. Our intention is to indicate how each department is likely to deliver a positive all-round experience to future students, and in order to assess this we refer to how past students in the department have fared. We quantify the resources and staff contact that have been dedicated to past students, we look at the standards of entry and the likelihood that students will be supported to continue their studies, before looking at how likely students are to be satisfied, to exceed expectations of success and to have positive outcomes after completing the course. Bringing these measures together, we get an overall score for each department and rank departments against this.
For comparability, the data we use focuses on full-time first degree students. For those prospective undergraduates who do not know which subject they wish to study but still want to know where institutions rank in relation to one another, the Guardian scores have been averaged for each institution across all subjects to generate an institution-level table. –
Changes introduced for 2021
The structure and methodology of the rankings has remained broadly constant since 2008 but there have been some significant changes to the data used in this year’s guide, causing some big changes throughout the rankings.
Career prospects
Most significant is the survey from which we derive our career prospects score. We still value the same things – graduates entering professional, managerial and technical occupations, or entering HE or professional further study – but the nature of the survey has fundamentally changed. Previously we drew on results from the DLHE survey (Destinations of Leavers from HE) which operated 6 months after graduation and was conducted by institutions contacting their own students. The new instrument that replaces it, the Graduate Outcomes survey, uses a central agency to contact graduates 15 months after graduation.
With its later census point, contact details perish and response rates for the new survey are lower than they were for the DLHE. We have reduced the threshold at which we use data from the survey from 25 respondents to 20 in order to mitigate the increased incidence of departments missing this information. Due to data protection rules, we only display the percentages with positive graduate outcomes for departments that had 22.5 or more respondents.
Data from the graduate outcomes data has been used by HESA to produce experimental statistics and one judgement we had to make was how to regard students who had entered further study, completed it, and were then unemployed at the census point. Subject to the interim study being at a professional or HE level, we have treated such cases as representing a positive student outcome. We also chose to include partial survey completions subject to having complete information about the activities occupying the respondent.
The National Student Survey
Each year the National Student Survey (NSS) is conducted between January and April to gauge the satisfaction levels of final-year students. We normally use results from the previous year but our later publication date meant that results from the 2020 survey had become available.
Analysis by the Office for Students, who commission the NSS, was extensive and contained no reason to not use the data in the guide. Only 21% of responses came after the Covid-19 global pandemic was declared by the WHO and there was no evidence to suggest that results had been distorted or invalidated by the outbreak.
We usually use the most recent year of NSS results and resort to two-year averages when results are based on a small number of responses (under 23) , or show erratic year-on-year changes while being based on a population of under 90.
This year we have averaged the results over the two most recent surveys in all cases where there was consistent data availability, and always used the results if the 2020 survey had 23 respondents or more. Where the number of 2020 respondents is under 23 but the total across 2019 and 2020 is above 23 we have usually used the aggregated results, but excluded results that were inconsistent and gave an unreliable impression of future student satisfaction levels.
Standardisation
Despite efforts to maintain the number of departments for which we can use career prospects data, there are subjects in which very few departments have a valid and reliable score. In order to build up a total score for each department, we normally standardise each metric in relation to the distribution of scores within the subject. When the number of departments with a valid metric score has fallen beneath nine, we now standardise in relation to a broader subject group.
Details of each metric
Entry standards
This measure seeks to approximate the aptitude of fellow students who a prospective student can anticipate and reports the observed average grades of students joining the department – not the conditions of admission to the course that may be advertised. Average tariffs are determined by taking the total tariff points of first-year, first degree, full-time entrants who were aged under 21 at the start of their course, if the qualifications that they entered with could all be expressed using the tariff system devised by UCAS. There must be more than seven students in any meaningful average and only students entering year one of a course (not a foundation year) with certain types of qualification are included. Departments that are dominated by mature entrants are not considered appropriate for this statistic because the age filter would capture and represent the entry tariff of only the minority of students.
This metric contributes 15% to the total score of a department. It is released at HESA cost centre level, and we map each cost centre to one or more of our subjects.
Student-staff ratios
Student-staff ratios (SSR) seek to approximate the levels of staff contact that a student can expect to receive by dividing the volume of students who are taking modules in a subject by the volume of staff who are available to teach it. Thus a low ratio is treated positively – it indicates that more staff contact could be anticipated.
Staff and students are reported on a ‘full-time equivalent’ basis and research-only staff are exclude from the staff volume. Students on placement or on a course that is franchised to another provider have their volume discounted accordingly.
At least 28 students and three staff (both FTE) must be present in an SSR calculation using 2018-19 data alone. Smaller departments that had at least seven student and two staff FTE in 2018-19, and at least 30 student FTE in total across 2018-19 and 2017-18, have a two-year average calculated.
This metric contributes 15% to the total score of a department. It is released at HESA cost centre level, and we map each cost centre to one or more of our subjects.
Expenditure per student
In order to approximate the level of resources that a student could expect to have dedicated to their provision, we look at the total expenditure in each subject area and divide it by the volume of students taking the subject. We exclude academic staff costs as the benefits of high staff volumes are already captured by the student-staff ratios but recognise that many costs of delivery are centralised: we add the amount of money each provider has spent per students on academic services such as libraries and computing facilities per student, over the past two years.
This metric is expressed as points/10 and contributes 5% to the total score of a department.
Continuation
Taking a degree-level course is a positive experience for most students but is not suited to everybody and some students struggle and discontinue their studies. Providers can do a lot to support their students and to promote engagement with studies and with the broader higher education experience, and this measure captures how successful each department is in achieving this. We look at the proportion of students who continue their studies beyond the first year and measure the extent to which this exceeds expectations based on entry qualifications.
To achieve this, we take all first year students on full-time first degree courses that are scheduled to take longer than a year to complete, and look ahead to the first of December in the following academic year to observe the proportion who are still active in higher education. This proportion is viewed positively, regardless of whether the student has switched course, transferred to a different provider, or been required to repeat their first year – only those who are inactive in the UK’s HE system are counted negatively.
To take the effect of entry qualifications into account we create an index score for each student who has a positive outcome, using their expectation of continuation up to a maximum of 97%. To calculate the score there must have been 35 entrants in the most recent cohort and 65 across the past two or three years.
This index score, aggregated across the last two or three years, contributes 10% to the total score of non-medical departments. However, it is the percentage score – also average over two or three years - that is displayed.
Student satisfaction
The National student survey asks final year students for the extent to which they agree with 27 positive statements about their academic experience of the course and support that they received. Responses are on a 5-point Likert scale (1. definitely disagree to 5. definitely agree) and we take the responses from full-time first degree students registered at the provider course to produce two statistics: a satisfaction rate and an average response. The satisfaction rate looks across the questions concerned and reports the proportion of responses that were ‘definitely agree’ or ‘mostly agree’, while the average response gives the average Likert score between 1 and 5 that was observed in the responses to those questions.
To assess the teaching quality that a student can expect , we took responses from the 2019 and 2020 NSS surveys and aggregated them for the following questions:
Staff are good at explaining things.
Staff have made the subject interesting.
The course is intellectually stimulating.
My course has challenged me to achieve my best work.
The overall satisfaction rate for each provider is displayed, and the average response is used with a 10% weighting.
To assess the likelihood that a student will be satisfied with the feedback they receive we took responses from the 2019 and 2020 NSS surveys and aggregated them for the following questions:
The criteria used in marking have been clear in advance.
Marking and assessment has been fair.
Feedback on my work has been timely.
I have received helpful comments on my work.
The overall satisfaction rate for each provider is displayed, and the average response is used with a 10% weighting.
To assess the overall satisfaction of students with their courses we aggregated responses from the 2019 and 2020 NSS surveys for the following statement: “Overall, I am satisfied with the quality of the course.�?
The overall satisfaction rate for each provider is displayed, and the average response is used with a 5% weighting.
Data was released at the CAH (common aggregation hierarchy) levels of aggregation and we used details of how these map to JACS (Joint academic classification of subjects) to weight and aggregate results for each of our 54 subjects, prioritising results from the most granular level.
Value-added scores
In order to assess the extent to which each department will support its students towards achieving good grades, we use value-added scores to track students from enrolment to graduation. A student’s chances of getting a good classification of degree (a first or a 2:1) are already affected by the qualifications that they start with, so our scores take this into account and report the extent to which a student exceeded expectations.
Each full-time student is given a probability of achieving a first or 2:1, based on the qualifications that they enter with or, if they have vague entry qualifications, the total percentage of good degrees expected for the student in their department. If they manage to earn a good degree, then they score points that reflect how difficult it was to do so (in fact, they score the reciprocal of the probability of getting a first or 2:1). Otherwise they score zero. Students taking integrated masters are always regarded as having a positive outcome.
At least 30 students must be in a subject for a meaningful value-added score to be calculated using the most recent year of data alone. If there are more than 15 students in the most recent year and the total number across two years reaches 30, then a two-year average is calculated.
This metric is expressed as points/10 and contributes 15% to the total score of a department, except in medical subjects where a variant version is worth 5%.
Career prospects
Using results from the new Graduate Outcomes survey for the graduating cohort of 2017-18, we seek to assess the extent to which students have taken a positive first step in the 15 months after graduation, in the hope that similar patterns will repeat for future cohorts. We value students that enter graduate level occupations (approximated by SOC groups 1-3: professional, managerial & technical occupations) and students that go on to further study at a professional or HE level and treat these students as positive.
Students report one or more activities, and for each of these give more detail. If students are self-employed or working for an employer, we treat them as positive if the occupation is in SOC groups 1-3; if they have either finished a course or are presently taking one then we look at the level and treat them positively accordingly. Students who have no activity that is regarded positively but who either reported that they were unable to work, or only partially completed the survey leaving details of an activity incomplete, are excluded from the metric.
The metric refers only to students who graduated from full-time first degree courses and we only use results if more than 20 students in a department responded. If between 20 and 22.5 responded we use the result but round or obscure the exact figure for data protection reasons.
In any year we avoid averaging results across years for this metric because the national economic environment that leavers find themselves in can have such a big effect on employment. This year the profound differences between DLHE and the graduate outcomes survey mean that we are never mixing results across years.
This metric is worth 15% of the total score in all the non-medical subjects.
Using metric results
First of all, we determine if a department has enough data to support a ranking. Often individual metrics are missing and we seek to keep the department in the rankings where we can. An institution can only be included in the table if the weighting value of any indicators that are missing add up to 40% or less, and if the institution’s relevant department teaches at least 35 full-time first degree students. There must also be at least 25 students (FTE) in the relevant cost centre.
For those institutions that qualify for inclusion in the subject table, each score is compared to the average score achieved by the other institutions that qualify, using standard deviations to gain a normal distribution of standardised scores (S-scores). The standardised score for student-staff ratios is negative, to reflect that low ratios are regarded as better. We cap certain S-scores – extremely high NSS, expenditure and SSR figures – at three standard deviations. This is to prevent a valid but extreme value from exerting an influence that far exceeds that of all other measures.
Although we don’t display anything, we need to plug the gap left in the total score that is left by any missing indicators. We use a substitution that firstly looks for the corresponding standardised score in the previous year and then, if nothing is available, resorts to looking at whether the missing metric is correlated to general performance in that subject. If it is, the department’s performance in the other metrics is used – effectively assuming that it would have performed as well in the missing metric as it did in everything else. If not, the average score achieved by other providers of the subject is used.
Using the weighting attached to each metric, the standardised scores are weighted and totalled to give an overall institutional score (rescaled to 100) against which the departments are ranked.
The institutional ranking
The institutional table ranks institutions according to their performance in the subject tables, but considers two other factors when calculating overall performance.
First, the number of students in a department influences the extent to which that department’s total standardised score contributes to the institution’s overall score. And second, the number of institutions included in the subject table determines the extent to which a department can affect the institutional table.
The number of full-time undergraduates in each subject is expressed as a percentage of the total number of full-time undergraduates counted in subjects for which the institution is included within the subject table. For each subject, the number of institutions included within the table is counted and the natural logarithm of this value is calculated. The total S-score for each subject – which can be negative or positive – is multiplied by these two values, and the results are summed for all subjects, to give an overall S-score for each institution. Institutions are ranked according to this overall S-score, though the value displayed in the published table is a scaled version of this, that gives the top university 100 points and all the others a smaller (but positive) points tally.
Each institution has overall versions of each of the indicators displayed next to its overall score out of 100, but these are crude institutional averages that are otherwise disconnected from the tables and give no consideration to subject mix. Therefore these institutional averages cannot be used to calculate the overall score or ranking position.
The indicators of performance for value added and for expenditure per student are treated slightly differently, because they need to be converted into points out of 10 before being displayed. Therefore these indicators do read from the subject level tables, again using student numbers to create a weighted average.
Institutions that appear in fewer than eight subject tables are not included in the main ranking of universities.
Course directory
The KIS database of courses, to which institutions provide regular updates to describe courses that students will be able to apply for in future years, is the data source of the courses that we list under each department in each subject group.
We have associated each full-time course with one or more subject groups, based on the subject data associated with the courses. We gave institutions the freedom to adjust these associations with subjects and also to change details of the courses. We include courses that are not at degree level, even though such provision is excluded from the data used to generate scores and rankings. Due to timing of publication, an update to this data will take place in September.