Evaluating structured assessment of anaesthesiologists' non-technical skills

R M H G Jepsen; Gerhard Peter Dieckmann; L Spanager; H T Lyk-Jensen; L Konge; C Ringsted; D Østergaard

doi:10.1111/aas.12709

Evaluating structured assessment of anaesthesiologists' non-technical skills

R M H G Jepsen, Gerhard Peter Dieckmann, L Spanager, H T Lyk-Jensen, L Konge, C Ringsted, D Østergaard

9 Citations (Scopus)

Abstract

BACKGROUND: Non-technical skills (NTS) are essential for safe and efficient anaesthesia. Assessment instruments with appropriate validity evidence can be used to ensure that anaesthesiologists possess the NTS necessary to deliver high-standard patient care. The aims were to collect validity evidence using a contemporary validity framework for the assessment instrument Anaesthesiologists' Non-Technical Skills in Denmark (ANTSdk) regarding response process and internal structure (including reliability), and to investigate the effect of rater training on these properties.

METHODS: An explorative study was undertaken at the Danish Institute for Medical Simulation, Copenhagen, Denmark. In a 1-day session, using ANTSdk, a convenience sample of 19 anaesthesiologists rated trainee anaesthesiologists' NTS in nine video-recorded simulation scenarios before and after a 3-h training session.

RESULTS: Response process evidence: participants considered ANTSdk useful and feasible for NTS assessment. Internal structure evidence: inter-rater reliability (single measures) largely expressed substantial agreement (ICC ≥ 0.55 and ICC ≥ 0.60 for pre- and post-training ratings respectively). Strong internal consistency of ratings was found (Spearman's correlation coefficient ≥ 0.82). Accuracy of participants' ratings compared with reference ratings (± 1 scale point) was notable (76% and 78% for pre- and post-training ratings, respectively). The results indicate that the elements 'Demonstrating self-awareness', 'Reassessing decisions', 'Assessing competencies', and 'Supporting others' need more attention in future rater training.

CONCLUSION: The validity evidence collected on content, response process, and internal structure, suggests that ANTSdk is easy to use on video-recorded simulation scenarios, indicating that ANTSdk is a feasible instrument for NTS assessment during anaesthesia training.

Original language	English
Journal	Acta Anaesthesiologica Scandinavica
Volume	60
Issue number	6
Pages (from-to)	756-766
Number of pages	11
ISSN	0001-5172
DOIs	https://doi.org/10.1111/aas.12709
Publication status	Published - 1 Jul 2016

Keywords

Anesthesiologists
Anesthesiology
Clinical Competence
Denmark
Female
Humans
Male
Reproducibility of Results

Access to Document

10.1111/aas.12709

Cite this

@article{f18d3239ec844de1b86a2709a58c16c5,

title = "Evaluating structured assessment of anaesthesiologists' non-technical skills",

abstract = "BACKGROUND: Non-technical skills (NTS) are essential for safe and efficient anaesthesia. Assessment instruments with appropriate validity evidence can be used to ensure that anaesthesiologists possess the NTS necessary to deliver high-standard patient care. The aims were to collect validity evidence using a contemporary validity framework for the assessment instrument Anaesthesiologists' Non-Technical Skills in Denmark (ANTSdk) regarding response process and internal structure (including reliability), and to investigate the effect of rater training on these properties.METHODS: An explorative study was undertaken at the Danish Institute for Medical Simulation, Copenhagen, Denmark. In a 1-day session, using ANTSdk, a convenience sample of 19 anaesthesiologists rated trainee anaesthesiologists' NTS in nine video-recorded simulation scenarios before and after a 3-h training session.RESULTS: Response process evidence: participants considered ANTSdk useful and feasible for NTS assessment. Internal structure evidence: inter-rater reliability (single measures) largely expressed substantial agreement (ICC ≥ 0.55 and ICC ≥ 0.60 for pre- and post-training ratings respectively). Strong internal consistency of ratings was found (Spearman's correlation coefficient ≥ 0.82). Accuracy of participants' ratings compared with reference ratings (± 1 scale point) was notable (76% and 78% for pre- and post-training ratings, respectively). The results indicate that the elements 'Demonstrating self-awareness', 'Reassessing decisions', 'Assessing competencies', and 'Supporting others' need more attention in future rater training.CONCLUSION: The validity evidence collected on content, response process, and internal structure, suggests that ANTSdk is easy to use on video-recorded simulation scenarios, indicating that ANTSdk is a feasible instrument for NTS assessment during anaesthesia training.",

keywords = "Anesthesiologists, Anesthesiology, Clinical Competence, Denmark, Female, Humans, Male, Reproducibility of Results",

author = "Jepsen, {R M H G} and Dieckmann, {Gerhard Peter} and L Spanager and Lyk-Jensen, {H T} and L Konge and C Ringsted and D {\O}stergaard",

note = "{\textcopyright} 2016 The Acta Anaesthesiologica Scandinavica Foundation. Published by John Wiley & Sons Ltd.",

year = "2016",

month = jul,

day = "1",

doi = "10.1111/aas.12709",

language = "English",

volume = "60",

pages = "756--766",

journal = "Acta Anaesthesiologica Scandinavica",

issn = "0001-5172",

publisher = "Wiley-Blackwell",

number = "6",

}

TY - JOUR

T1 - Evaluating structured assessment of anaesthesiologists' non-technical skills

AU - Jepsen, R M H G

AU - Dieckmann, Gerhard Peter

AU - Spanager, L

AU - Lyk-Jensen, H T

AU - Konge, L

AU - Ringsted, C

AU - Østergaard, D

PY - 2016/7/1

Y1 - 2016/7/1

N2 - BACKGROUND: Non-technical skills (NTS) are essential for safe and efficient anaesthesia. Assessment instruments with appropriate validity evidence can be used to ensure that anaesthesiologists possess the NTS necessary to deliver high-standard patient care. The aims were to collect validity evidence using a contemporary validity framework for the assessment instrument Anaesthesiologists' Non-Technical Skills in Denmark (ANTSdk) regarding response process and internal structure (including reliability), and to investigate the effect of rater training on these properties.METHODS: An explorative study was undertaken at the Danish Institute for Medical Simulation, Copenhagen, Denmark. In a 1-day session, using ANTSdk, a convenience sample of 19 anaesthesiologists rated trainee anaesthesiologists' NTS in nine video-recorded simulation scenarios before and after a 3-h training session.RESULTS: Response process evidence: participants considered ANTSdk useful and feasible for NTS assessment. Internal structure evidence: inter-rater reliability (single measures) largely expressed substantial agreement (ICC ≥ 0.55 and ICC ≥ 0.60 for pre- and post-training ratings respectively). Strong internal consistency of ratings was found (Spearman's correlation coefficient ≥ 0.82). Accuracy of participants' ratings compared with reference ratings (± 1 scale point) was notable (76% and 78% for pre- and post-training ratings, respectively). The results indicate that the elements 'Demonstrating self-awareness', 'Reassessing decisions', 'Assessing competencies', and 'Supporting others' need more attention in future rater training.CONCLUSION: The validity evidence collected on content, response process, and internal structure, suggests that ANTSdk is easy to use on video-recorded simulation scenarios, indicating that ANTSdk is a feasible instrument for NTS assessment during anaesthesia training.

AB - BACKGROUND: Non-technical skills (NTS) are essential for safe and efficient anaesthesia. Assessment instruments with appropriate validity evidence can be used to ensure that anaesthesiologists possess the NTS necessary to deliver high-standard patient care. The aims were to collect validity evidence using a contemporary validity framework for the assessment instrument Anaesthesiologists' Non-Technical Skills in Denmark (ANTSdk) regarding response process and internal structure (including reliability), and to investigate the effect of rater training on these properties.METHODS: An explorative study was undertaken at the Danish Institute for Medical Simulation, Copenhagen, Denmark. In a 1-day session, using ANTSdk, a convenience sample of 19 anaesthesiologists rated trainee anaesthesiologists' NTS in nine video-recorded simulation scenarios before and after a 3-h training session.RESULTS: Response process evidence: participants considered ANTSdk useful and feasible for NTS assessment. Internal structure evidence: inter-rater reliability (single measures) largely expressed substantial agreement (ICC ≥ 0.55 and ICC ≥ 0.60 for pre- and post-training ratings respectively). Strong internal consistency of ratings was found (Spearman's correlation coefficient ≥ 0.82). Accuracy of participants' ratings compared with reference ratings (± 1 scale point) was notable (76% and 78% for pre- and post-training ratings, respectively). The results indicate that the elements 'Demonstrating self-awareness', 'Reassessing decisions', 'Assessing competencies', and 'Supporting others' need more attention in future rater training.CONCLUSION: The validity evidence collected on content, response process, and internal structure, suggests that ANTSdk is easy to use on video-recorded simulation scenarios, indicating that ANTSdk is a feasible instrument for NTS assessment during anaesthesia training.

KW - Anesthesiologists

KW - Anesthesiology

KW - Clinical Competence

KW - Denmark

KW - Female

KW - Humans

KW - Male

KW - Reproducibility of Results

U2 - 10.1111/aas.12709

DO - 10.1111/aas.12709

M3 - Journal article

C2 - 26988291

SN - 0001-5172

VL - 60

SP - 756

EP - 766

JO - Acta Anaesthesiologica Scandinavica

JF - Acta Anaesthesiologica Scandinavica

IS - 6

ER -

Evaluating structured assessment of anaesthesiologists' non-technical skills

Abstract

Keywords

Access to Document

Fingerprint

Cite this