Автор неизвестен - Mededworld and amee 2013 conference connect - страница 42
Validity of the Multiple Mini-Interview (MMI)
Michael Dodson (University of Notre Dame Australia, Melbourne Clinical School, 300 Princes Hwy, Werribee 3030, Australia)
performance on a selection MMI. The Deakin MMI, comprises ten stations of eight minutes duration, that assess qualities likely to underpin effective and professional encounters with patients and colleagues, maintenance of professional standards and social responsibility.
Summary of work: This study examined the usefulness of the MMI in predicting student performance during the Deakin Medical Course (DMC), a 4 year graduate program that incorporates 4 key themes; Doctor and Patient (DP); Knowledge of Health and Illness (KHI); Doctors, Cultures, Peoples and Institutions (DPCI); and, Ethics, Law and Professional Development (ELPD). Summary of results: No statistically significant correlations were observed between individual MMI stations or total MMI score and performance in the four themes of the DMC, or the course overall. MMI score did, however, predict performance in the second year of the course (r=0.45, p=0.01). Of the three selection tools, GPA best predicted overall performance during the medical course (r=0.29, p=0.12). Combining GPA with MMI score improved predictive validity for overall performance (r=0.49, p=0.07), suggesting that 25% of variation in performance during the medical course may be accounted for by the combination of GPA and MMI score.
Conclusions: Although the study was limited by small participant numbers, the results suggest selection tools that provide information about cognitive and non-cognitive qualities may provide complementary information about medical school performance. Take-home messages: Together, GPA and MMI score were particularly useful in predicting performance in the
The impact on reliability of MMIs by using Skype and reducing the number of stations to 5
Deborah O'Mara (Sydney Medical School, Assessment Unit Office of Medical Education, Room 108 Edward Ford Building, Corner of Fisher and Physics Roads, The University of Sydney, Camperdown NSW 2006, Australia) Imogene Rothnie (Sydney Medical School, Assessment Unit Office of Medical Education, Sydney, Australia)
Background: The Sydney Medical School conducts MMIs as part of its selection process. Two major policy changes have been introduced in recent years. Firstly, in 2011 Skype was introduced for conducting MMI
ABSTRACT BOOK: SESSION 4 MONDAY 26 AUGUST: 1400-1530
interviews with international candidates while all local candidates continued to be interviewed in person. Secondly, the number of MMI stations was reduced from 9 to 5 stations in 2012 with 24 circuits. Summary of work: Variance components analysis was used to assess the differences in reliability between 9 and 5 stations for the two methods of interviewing. Summary of results: The reliability for the Skype MMI was found to be higher than the in-person MMI for 9 stations in 2011 and 5 stations in 2012; the Generalisability coefficient for Skype was 0.76 and 0.70 for 2011 and 2012 respectively and for in-person local interviews 0.70 and 0.63 for 2011 and 2012. While the reliability of 5 stations was less than that for 9 stations, the sources of variation changed in that there was less error variance and less variance due to the MMI question but more variance due to the interaction of candidate and interviewer.
Conclusions: The reduction in reliability from 9 to 5 MMI stations is not sufficient to threaten the reliability of the selection process, when the greatest sources of variation are candidates and interviewers and their interaction.
Take-home messages: Skype provides a cost efficient and reliable means of conducting MMI interviews and a reliable judgement can be made with as few as 5 MMI stations.
Multiple mini interviews: checklist scores, global ratings or both?
Godfrey Pell (University of Leeds, Leeds Institute of Medical Education, School of Medicine, Room 7.09,
Worsley Building, LIME Level 7, Leeds LS2 9NL, United
Background: In the health care professions, especially medicine and dentistry, the undergraduate programmes at most universities are over-subscribed, and it is therefore necessary to develop an objective and defensible selection process. Summary of work: This work investigates the recruitment process for two different health care professional programmes (medicine and dentistry) using different models; one model using both a checklist and global rating, the other a global rating alone. Summary of results: Both checklist and global ratings discriminated well between candidates with a high degree of internal consistency. In the model using both checklists and global ratings there was a high level of correlation between the two measures (R>0.8), but there were some interesting outliers when both rankings were compared, i.e. some candidates impressed assessors but had low checklist scores and vice versa. Although an outline of the stations soon appeared on the web, there was no noticeable improvement in performance over time for candidates who might have seen this material. Conclusions: Whilst both models discriminate between candidates, the treatment of those who would be selected on one measure but not on the other is a key
issue to be resolved. Also, should station profile play any part in the selection process? Whilst we approve of the general principle of parsimony, ie keeping things simple, we feel that the additional information given by the triangulation of assessment data is both significant and useful.
Take-home messages: As with OSCE stations, basing high level decisions on a single measure or score is not advisable.
The utility of Multi Faceted Rasch Modelling to evaluate applicant ability, examiner severity and consistency, and item difficulty in high stakes multiple mini interview for selection into graduate entry medicine
Imogene Rothnie (University of Sydney, Office of Medical Education, Sydney, Australia) Chris Roberts (University of Sydney, Sydney Medical Program, Charles Perkins Centre, A14 - Quadrangle, Camperdown 2006, Australia)
Background: In overcoming recognized limitations of Generalisability Theory, Multi-Faceted Rasch Modelling (MFRM) provides individual metrics of student ability, examiner severity, and item difficulty, independently of the sample undertaking the assessment. We report an example applied to a high stakes Multiple-Mini-Interview (MMI) for selection into graduate entry medicine
Summary of work: 705 Applicants, 179 interviewers and 33 MMI questions were organized into 24 circuits of 5 stations. MFRM was applied to the data to investigate whether the assessment could be considered to target a unidimensional construct, if interviewers demonstrated significantly different levels of severity, and if circuits of questions could be considered equivalent in terms of difficulty.
Summary of results: The MMI measured a unidimensional construct, and candidate ability was the largest single contributor to total scores, contributing 30% variation between scores. There was a significant difference in the levels of interviewer severity (p<0.05). There was no significant difference in difficulty between circuits. The impact on the rank list of applicants is demonstrated, where MMI scores for the severity of the interviewer have been adjusted. Conclusions: MFRM extends the understanding of the psychometrics of observed performances in medical education.
Take-home messages: MFRM is a useful approach to advancing knowledge about observed performance assessments and provides a metric, which educators could use to adjust candidate scores for construct irrelevant variance such as examiner severity. Further work needs to be undertaken to establish the validity of the single construct measured by the MMI.
4M Short Communications: Virtual
Location: Club D, PCC
Research in the Gamification of Medical Education
Adrian Raudaschl (Greater Glasgow and Clyde, Medicine, Glasgow, United Kingdom)
Background: Gamification of medical education aims to apply the underlying principles that make games engaging and enjoyable to a test based format. Cognitive psychology has shown repeated testing of information promotes better retention and comprehension of information. I identified game design elements such as attractive interfaces, scoring systems, awards, story narrative, short and long term feedback, dynamic objectives and competition with the aim to encourage repeated voluntary testing of information. Summary of work: We designed a range of interactive virtual patient cases for the iPhone platform in collaboration with Glasgow Medical School. A series of multiple choice questions (MCQ's) were then derived from main learning objectives (e.g. investigations, treatments). Cases would gradually progress from presentation to their therapeutic conclusions; the aim to teach medical knowledge and demonstrate its direct clinical application.
Summary of results: The format was similar to that of an interactive video game. Our first application 'Ward Round' has been rated highly and downloaded over 1,300 times. For our app 'Microbiology", feedback from 51 students showed the app helped increase their knowledge of microbiology and 73% felt using the app was a good use of study time. Conclusions: Students have highly rated our gamification approach to medical education and find it a useful resource. This project has added value to our current medical school module on microbiology, and opens opportunities to introduce similar applications to other areas of the curriculum.
Take-home messages: Use of smart phones in medical education is an exciting new field, and the opportunities for innovative application are just starting to begin.
Expectations and experiences of the collaborative aspect of virtual patient work
Samuel Edelbring (Karolinska Institutet, LIME, Tomtebodavagen 18A, Stockholm SE-171 77, Sweden) Ingrid E Lundberg (Karolinska Institutet, Dept. of Medicine, Stockholm, Sweden) Nabil Zary (Karolinska Institutet, LIME, Stockholm, Sweden)
Background: The small group setting has proved beneficial for learning (Lou et al., 2001). Furthermore, the reasoning process is a goal in itself within an
ABSTRACT BOOK: SESSION 4 MONDAY 26 AUGUST: 1400-1530
interpersonal view of learning (Crook, 2010). In medical education the use of virtual patients (VPs) has emerged as a method to train clinical reasoning which implies an increased interest for peer collaboration. The VPs are however often used individually in a self-study manner. One study reports 94% individual use (Fall et al., 2005). The aim of this study is to gain knowledge about how students perceive collaboration using VPs and the reasons for individual or collaborative VP work. Summary of work: Students worked with four VPs during clinical clerkship in Rheumatology. All students during one semester (n=30) were asked to fill in a questionnaire concerning collaborative VP work. Questionnaire responses were analysed thematically (Braun and Clarke, 2006).
Summary of results: Twenty-nine students (97%) answered the questionnaire. 62% of the students worked individually with the virtual patients, 24% both individually and collaboratively, and 14% only in the collaborative setting. All of those working in the collaboratively stated learning reasons for doing so. Almost all (94%) of the reasons for working individually were of practical character. Those working in both settings stated both practical and learning reasons for doing so.
Conclusions: Students clearly see educational benefits of working collaboratively with VPs. However, for practical reasons collaborative case work seems often to give way to the individual self-study approach. Take-home messages: Practical opportunities should be arranged for in order to benefit from collaborative learning when using VPs.
Can virtual patients be shared internationally?
Kieran McGlade (Queen's University Belfast, Centre for Medical Education, Department of General Practice, Dunluce Health Centre, 1 Dunluce Avenue, Belfast BT9
7HR, United Kingdom)
Kati Hakkarainen (University of Tampere, School of Medicine, Tampere, Finland)
David McCarthy (Queen's University Belfast, Centre for Medical Education (student), Belfast, United Kingdom) Richard Plumb (Queen's University Belfast, Centre for Medical Education, Belfast)
Gerard Gormley (Queen's University Belfast, Centre for Medical Education, Belfast, United Kingdom) Ciaran OGorman (Ulster Hospital, Critical Care Complex Laboratories, Belfast, Turkmenistan)
Background: The Medical schools at Queens University Belfast and University of Tampere piloted the sharing of Virtual Patient (VP) material.
Summary of work: Two VPs authored by the teachers in Belfast were used as revision material in an integrated PBL block dealing with infectious diseases in Tampere. The VPs were created using Riverside software shared by members of the International Virtual medical School (IVIMEDS). The VPs consisted of descriptions of patient cases, multimedia commentary, multiple choice and open questions with immediate feed-back and scoring.
Five VPs were reviewed in Tampere in relation to the block and two immediately matched specific curricular objectives. Minimal editing was required consisting of altering the terms of medical practices to meet Finnish circumstances, changing the linked guidelines to local ones and also changing a small amount of text, two MCQs and three open questions. This was easily accomplished using Riverside.
Summary of results: The VPs were published in Moodle VLE during the last week of the block as optional study resources. Students were asked to optionally return the completed VPs as PDFs. According to log-in data 59 students out of 98 used the VPs, Fourteen PDFs were returned. Students were very positive about the Belfast VPs in their regular online feedback forms. The content was regarded as interesting and the questions with synchronous feedback suitably challenging. Conclusions: Although sharing was not considered during design, the VPs were easily adapted. This process could be even more straight forward with advance planning.
Take-home messages: Sharing of eLearning materials internationally is very feasible.
The Generation 4.5 Project: Using Ethical Virtual Patients in Teaching
Carwyn Hooper (St George's, University of London, Division of Population Health Science & Education, Cranmer Terrace, London SW17 0RE, United Kingdom) Terry Poulton (St George's, University of London, Division of Population Health Science & Education, London, United Kingdom)
Sheetal Kavia (St George's, University of London, Division of Population Health Science & Education, London, United Kingdom)
Bryan Vernon (Newcastle University, Newcastle, United Kingdom)
Pirashanthie Vivekananda-Schmidt (The University of Sheffield, United Kingdom)
Georgia Testa (University of Leeds, United Kingdom)
Background: Use of Virtual Patients (VP) is well established in medical education. They improve clinical reasoning, decision making skills and knowledge retention through engaging and interactive learning experiences. The St George's team devised the world's first Ethical Virtual Patient (EVP) cases with JISC support. Summary of work: 5 EVP cases were placed on a secure website and sent to the 20 participating medical schools with links to student and staff evaluation questionnaires. Open and closed questions were used. Questionnaire data was complemented by focus group interviews at selected schools.
Summary of results: The evaluation occurred between 1st May 2012-1st May 2013. Initial data suggests EVP cases are popular with students who find them engaging, stimulating and interesting. These findings corroborate previous work (Ref). Staff found EVPs easy to use and useful for contextualising teaching.
ABSTRACT BOOK: SESSION 4 MONDAY 26 AUGUST: 1400-1530
Conclusions: VPs have been used in medical education to good effect for many years. Surprisingly EVPs have not previously been used to improve medical ethics teaching. Our multi-site data suggests that EVPs can be utilised to improve ethics teaching by: (1) promoting student interaction; (2) consideration of alternative arguments; (3) role play decisions in a safe environment. Take-home messages: EVPs represent a new and potentially game-changing pedagogical tool for those who teach ethics, law, and professionalism to medical students. Our research suggests that students and staff find EVPs stimulating and useful: they enable students to test out ethical decision making.
Exploring the validity and reliability of a questionnaire for evaluating virtual patient design with a special emphasis on fostering clinical reasoning
Soren Huwendiek (Institute of Medical Education, Bern
University, Department of Assessment and Evaluation,
Konsumstr. 13, Bern 3010, Netherlands)
Bas deLeng (Maastricht University, Department of
Educational Development and Research, Maastricht,
Cees van der Vleuten (Maastricht University, Department of Educational Development and Research, Maastricht, Netherlands)
Georg F Hoffmann (University Children's Hospital
Heidelberg, Clinic I, Heidelberg, Germany)
Burkhard Tonshoff (University Children's Hospital
Heidelberg, Clinic I, Heidelberg, Germany)
Diana Dolmans (Maastricht University, Department of
Educational Development and Research, Maastricht,
Background: The design of Virtual Patients (VPs) is essential. So far there are no validated evaluation instruments for VP design published. Summary of work: We examined three sources of validity evidence of an instrument to be filled out by students aimed at measuring the quality of VPs with a special emphasis on fostering clinical reasoning: (1) Content was examined based on theory of clinical reasoning and an international VP expert team. (2) Response process was explored in think aloud pilot studies with students and content analysis of free text questions accompanying each item of the instrument. (3) Internal structure was assessed by confirmatory factor analysis (CFA) using 2547 student evaluations and reliability was examined utilizing generalizability analysis.
Summary of results: Content analysis was supported by theory underlying Gruppen and Frohna's clinical reasoning model on which the instrument is based and an international VP expert team. The pilot study and analysis of free text comments supported the validity of the instrument. The CFA indicated that a three factor model comprising 6 items showed a good fit with the data. Alpha coefficients per factor were 0,74 - 0,82. The findings of the generalizability studies indicated that 40-
200 student responses are needed in order to obtain reliable data on one VP.
Conclusions: The described instrument has the potential to provide faculty with reliable and valid information about VP design.
Take-home messages: We present a short instrument which can be of help in evaluating the design of VPs.
The development of a virtual city for the environment of a Primary Health Care Course in the Amazonian region
Alessandra Dahmer (Federal University of Health Sciences, Education information and health, Porto Alegre, Brazil)
Maria Eugenia Pinto (Federal University of Health Sciences, Department of Public Health, Porto Alegre, Brazil)
Fabricio Costa (Federal University of Health Sciences, Department of Public Health, Porto Alegre, Brazil) Otavio DAvila (Federal University of Rio Grande do Sul, Department of Public Health, Sarmento Leite, 245, Demetrio Ribeiro, 499, Porto Alegre 90010310, Brazil)
Background: The post-graduation course in Family's Health of the UNA-SUS/UFCSPA, developed in the modality of distance education, uses a virtual city where there are complex clinical cases that describe ordinary situations in the primary health care (PHC) in the Brazilian Amazon.
Summary of work: In order to adjust the course to the professionals who work in the Brazilian Amazon, it was necessary to develop an Amazonian virtual city that matched the regional endemic specificities. For the creation and development of a Muiraquitan virtual city, a multidisciplinary team was composed in order to guide the work. The conception of the city, started with a meeting on the Para's state, which belongs to the Amazonian region, bringing together the UFCSPA. In this phase, it was possible to make an initial recognition of the area, the habits and the needs of the population, as well as the perspectives of the local professionals. Summary of results: The social, demographic and geographic characteristics of the Amazonian virtual city were defined in order to adapt to a new environment six new clinical complex cases that describe ordinary endemic situation, considering the primary health care of that region.
Conclusions: The virtual city brought near the complex cases and the cultural, social and economic reality of the Amazon and allowed simulation of the endemic situations that the students face in their practice in the
Take-home messages: The creation of a fictitious Amazonian city is an advance for the qualification of the course offered to the health professionals in a region that offers little training in this area.
4N Short Communications: Learning Online
Location: Meeting Room 2.1, PCC 4N/1
The whacky in the wiki - supporting online tutors
Janet MacDonald (Cardiff University, School of Postgraduate Medical and Dental Education, Heath Park, Cardiff CF14 4YS, United Kingdom) Lesley Pugsley (Cardiff University, School of Postgraduate Medical and Dental Education, Cardiff, United Kingdom)
Lynne Allery (Cardiff University, School of Postgraduate Medical and Dental Education, Cardiff, United Kingdom)
Background: In recent years there has been a significant shift to online provision within medical education courses, however, little attention has been given to the support that online tutors and facilitators require. Access to support in this role is crucial as online tutors are often geographically distant to the host university. On the e-postgraduate certificate, diploma and MSc in medical education at Cardiff UK, online tutors are supported through the use of a staff wiki to enable them to share ideas and strategies and also gain support and advice from peers. A wiki is an online collaborative tool that allows individuals to discuss, build and share content in an online environment. Summary of work: A new wiki is created each week during term time but online tutor use of the wiki in our setting has varied substantially with the reasons for this not being evident. This, therefore, became the focus for this particular investigation. The quantity, type and content of posts were analysed for each wiki during a four year period. Online tutors were also surveyed via questionnaires about their perception of the wiki, patterns of use and perceived benefits and difficulties. Summary of results: The key findings indicate that particular types of posts can have an inhibitive effect on other tutors participation in, and benefits from the wiki. In some instances this led to tutors disengagement with this online tool and, to a lesser extent, their contributions with students on the programme. Conclusions: Reasons for these particular type of posts are suggested and this exemplifies the need for ground rules and illustrates key pointers to be included. Take-home messages: Wikis can provide powerful support for online tutors but have the potential to undermine and inhibit contributions.