Ensuring medical graduates are competent is of central importance to a number of stakeholders including medical schools, faculty, regulators, employers, students and last but not least patients and their carers. Consequently, assessment of competence is one of the main areas of focus for medical education. Recognising the significance of excellence in student assessment the ASPIRE Board identified this as one of the initial themes. The aim of the ASPIRE initiative is to encourage and reward excellence. In this article the process of agreeing the criteria for excellence in assessment is described along with examples from successful applications.
Garantizar que los graduados de medicina sean competentes es una cuestión de vital importancia para las diversas partes interesadas, facultades de medicina, profesorado, autoridades reguladoras, proveedores de servicios, estudiantes y por último pero no por ello menos importante, los pacientes y sus cuidadores. En consecuencia, la evaluación de competencias supone una de las principales áreas de atención para la educación médica. Para los responsables de ASPIRE, una de las prioridades iniciales fue reconocer el significado de la excelencia en la evaluación del estudiante. El objetivo de la iniciativa ASPIRE es estimular y premiar la excelencia. En este artículo se describe el proceso empleado para acordar los criterios de reconocimiento de excelencia en evaluación junto con ejemplos que proceden de facultades premiadas.
Given the central importance of competency in medical graduates it is perhaps not surprising that assessment of students was chosen as one of the original themes for the ASPIRE programme (Fenoll-Brunet and Harden, 20151). Around the world medical schools have grappled with the problem of ensuring that students who are not yet at the level required to ensure efficient, effective and safe patient care are not let loose on an unsuspecting public. At the same time developments in our understanding of student testing and of the importance of assessment in driving learning has meant that this is an area that has gained a lot of attention in the medical education community. Virtually every medical education conference has a section on assessment and some conferences, for example the Ottawa conference, are wholly devoted to this area. Over time we have switched focus from merely testing knowledge to testing application of knowledge and the performance of skills both in controlled (for example in Objective Structured Clinical Examinations) and naturalistic working environments (for example using workplace based assessments).
The criteria for excellence in assessment were drawn up through a dephi-type process involving all the members of the assessment panel and took approximately three months to agree using several iterations. These initial criteria were used with a group of medical schools that volunteered for the initial ASPIRE pilot. Using this experience and based on pilot school feedback several changes were made to the criteria centred on improving the clarity of the requirements. Although evaluating the applications can prove challenging one of the major rewards of the review process is learning about some of the really creative innovations and system implementations in assessment in medical schools across the world. In the next part of this paper I will describe six areas of particular interest in the assessment theme.
Assessment for learningThe euphemism assessment drives learning is generally a truism and no matter what we want students to learn they commonly regard elements of the curriculum which are not assessed as unimportant. This notion of course is a powerful one and means that we can guide student learning by ensuring that students understand that this area of the course will be tested. Many years ago students had to guess what would be assessed and make personal judgments on what to learn hoping that these areas would come up in the exam. By being more explicit about the content of an assessment we can be sure that students would concentrate their learning on these areas.
The pace of the development of new knowledge and understanding in medicine demonstrates exponential growth. It is clear that doctors cannot hold all the information they require to practice in their heads. Treatment guidelines are constantly being updated in light of new research. Education needs to reflect this real world of practice and the idea of mastery of a subject has given way to a recognition of lifelong learning. Several applications have reflected this in that they looked to test only core knowledge in assessments. Information which is deemed core is assessed in written formats where memorisation is key such as single best answer formats. For areas of practice that require utlising the most up-to-date knowledge they used newer methods of assessments such as portfolios or research reports.
Assessment strategy and programmes of assessmentThe reviewers always looked for a clear assessment strategy, that is the school's overarching approach to assessment and how assessment fits within the wider curriculum. All the successful schools have demonstrated that they have such a programme of assessment. They provided evidence that the assessments were planned so that they built on each other and linked in a coherent way to the vision or mission of the school. Additionally, these schools reliably demonstrated that the students had achieved the desired learning outcomes and that the results of assessments not only informed further student learning but were also considered by Faculty to ensure that the school was a learning organisation itself. Reviewers also looked for schools to justify the choice of assessment tools and the timing and distribution of exams.
Using assessments as feedback for studentsAll of the schools who were scored excellent in their applications had detailed systems for providing as much feedback as possible to students about their performance, not only overall but also in the different domains of learning. Not only did these schools provide extensive feedback but also provided timely and effective support for students who had not performed as well as they might have done. Sometimes this support was in written format but mostly frequently it was in the form of face to face meetings with either personal tutors or course leaders. Those students who were required to re-sit assessments were provided with extra teaching or learning opportunities. In the case of practical skill development there were often intensive practice sessions to ensure the students had the best chance of success. All the schools also provided frequent formative assessments so that students and Faculty could ensure that individuals were on track to achieve competent practice.
Students as partnersMany schools had a declared purpose not only to engage students as partners in their learning but also in their assessment, for example as peer assessors in formative assessments. So that students not only wrote single best answer questions to examine knowledge in one school's formative assessments but in others students were supported in running formative OSCE sessions. In other schools’ senior students were encouraged and supported in developing on line assessment tools for junior years for example on-line self-assessment quizzes. Many schools had in depth discussions with the student body or their representatives about the types of assessments that were used and why they were chosen, how questions were developed and the methods that were adopted for standard setting together with the evidence for their use.
Assessment of professionalismIn the last decade it has become increasingly apparent that the assessment of student's professionalism is as important as assessing their knowledge and skills. Work by Maxine Papadakis,2,3 and Richard and Sylvia Cruess,4,5 and others have been key in this area. Medical schools realise that these are not one off assessments but part of an on-going assessment of professional attitudes and behaviour throughout a student's time at medical school. The assessment of professionalism is geared to detecting underlying recurrent patterns of unprofessional behaviour as opposed to one of errors of judgement.
Faculty reflection on their assessmentsThe way in which a medical school used information about student performance in assessments was also important to ensure that Faculty reviewed the quality of their own teaching and the student learning experience. Reviewers looked for evidence of systematic used of rigorous quality assurance systems which were used to continually improved the assessment programme. Feedback from students about the assessment process was a particular feature of excellent schools.
The Assessment theme of the ASPIRE programme is probably the most challenging area for school to achieve excellence. Currently only 23% of the applications have received the award of excellent. However, a number of applications which were not currently successful in gaining an overall award of excellence did have several areas of extremely good practice and these schools were strongly encouraged to undertake further development and reapply. The standards for excellence in this area are rightly exacting however, the assessment panel members hope that through providing encouragement and detailed feedback to applicants the ASPIRE initiative is supporting and contributing to the development of excellence in the area of student assessment in medical education.
Conflict of interestThe author has no conflicts of interest to declare.
