| Authors | Field | Aim | Design | Evidence based outcome of learning | Bias risk assessment based on individual MERSQI and NOS-E items (Granted score per item/Maximum potential score per item) | MERSQI total score | NOS-E total score | Impact level |
| | Music education Soprano recorder | To compare the effects of improvement focused versus fault focused corrective feedback on learning a music performance skill on a soprano recorder in elementary school and college students. | Non-randomized 2 group study
Setting: Half the trainees received improvement and half received fault focused corrective feedback. One of the researchers provided all feedback and intended to keep a neutral tone. Data collection: The researchers assessed the skills acquisition of trainees based on video-recordings through the amount of trials and time a trainee needed to independently perform in a duet. They also observed and noted the amount of corrections on what to improve, corrections on what went wrong and the amount of positive feedback that was provided. | Trainees in the improvement focused corrective feedback condition did not differ from trainees in the fault focused corrective feedback condition regarding number of trials and time in minutes {A} needed to perform independently. Observations of the applied teaching behavior (classification of behaviors in 3 predetermined categories by the researchers {B}) showed that improvement focused corrective feedback was higher in the improvement focused condition, fault focused corrective feedback was higher in the in the fault focused condition, but that the amount of positive feedback was equal in both conditions. | A: MERSQI: Non-randomized 2 group (2/3); Two institutions (1/1.5); Sampling response rate ≥ 75% (1.5/1.5); Objective data assessment (3/3); Validity evidence N/A; Data analysis beyond descriptive (2/2); Data analysis appropriate (1/1); Outcomes measuring skills acquisition (1.5/3). NOS-E: No representative group of participants (not randomly selected; 0/1); Comparable control group (1/1); Allocation of trainees not concealed (1/2); Study retention unlikely source of bias (1/1); No blinded assessment (0/1). B: MERSQI: Non-randomized 2 group (2/3); Two institutions (1/1.5); Sampling response rate ≥ 75% (1.5/1.5); Objective data assessment (3/3); No validity evidence established (0/3); Data analysis beyond descriptive (2/2); Data analysis appropriate (1/1); Outcomes measuring skills acquisition (1.5/3). NOS-E: No representative group of participants (not randomly selected; 0/1); Comparable control group (1/1); Allocation of trainees not concealed (1/2); Study retention unlikely source of bias (1/1); No blinded assessment (0/1). | A: 14.4* B: 12
M: 13.2 | A: 3 B: 3
M: 3 | 2 |
| | Surgical education Laparoscopic cutting task | To analyze whether an encouraging feedback style results in less subjective stress and better laparoscopic skills in inexperienced medical students than a harshly criticizing feedback style, a quiet observational feedback style or no feedback.
| Four group pre and post test (for the results relevant to our study)
Setting: Trainees were equally divided into four groups during simulation training, based on order of sign up: a control group (with no feedback); an observation only group (teacher only observing the trainee using minimal and neutral feedback); an encouraging feedback style group (teacher providing positive feedback and acting encouraging), and a harshly criticizing feedback style (teacher providing harsh criticism, being sarcastic and condescending). Trainees were encouraged to perform as quickly and accurate as possible. Trainees were naive to the study purpose. Data collection: The time to task completion was recorded and accuracy scores were collected and transformed into performance scores (based on the SAGES FLS scoring system; no further description). Performance scores were compared for the trainees’ first trial, last trial, and the mean performance score of all trials (ranging from 4 to 8) in between. | Trainees in all groups improved significantly during training. Trainees who were provided with harshly criticizing feedback behaviors scored lower than the trainees in the other groups on overall performance scores (time and accuracy scores transformed into a performance score based on the SAGES FLS scoring system; no further description {A}). Trainees provided with positive and encouraging feedback scored similar to trainees who received minimal and neutral feedback, and trainees who received no feedback. | A: MERSQI: Single group pre test post test (1.5/3); One institution (0.5/1.5); Sampling response rate ≥ 75% (1.5/1.5); Objective data assessment (3/3); Validity evidence N/A; Data analysis beyond descriptive (2/2); Data analysis appropriate (1/1); Outcomes measuring skills acquisition (1.5/3). NOS-E: No representative group of participants (sampling not described; 0/1); Comparable control group (1/1); Allocation of trainees not concealed (1/2); Study retention unlikely source of bias (1/1); No blinded assessment (0/1). | A: 13.2* | A: 3
| 2 |
| | Sport education Volleyball | To analyze whether a practice or command coaching style results in better volleyball sub skills performance in low, medium and highly skilled trainees. | Non-randomized 3 group cohort study
Setting: Based on a skills pre-test (unclear who took this) trainees were divided in high, medium and low skilled. Training lasted for 19 days. Per skills group, 50% was provided with the command and 50% with the practice coaching style by a professor in Physical education (no further description). Data collection: Data was collected on the pre-test, day 2 and 3 (midterm test), day 8 and 9 (midterm test) and day 17, 18 and 19 (post-test) according to the American Alliance for Health, Physical Education, Recreation and Dance test and the Stanley spike test (ratio correct to incorrect actions; no further description). Trainee performance was videotaped. Four observers (no further description) rated game play and practice trials. | Regarding practice trials, low skilled trainees in the command style condition performed better on volleyball set skills over the course of 19 days (ratio correct to incorrect performances {A}) and low skilled trainees in the practice style condition performed better on the volleyball spike skills (ratio correct to incorrect performances {B}). | A /B: MERSQI: Cohort study (2/3); One institution (0.5/1.5); Sampling response rate ≥ 75% (1.5/1.5); Objective data assessment (3/3); Content validity and internal validity established, no relationships to other variables (2/3); Data analysis beyond descriptive (2/2); Data analysis appropriate (1/1); Outcomes measuring skills acquisition (1.5/3). NOS-E: No representative group of participants (sampling not described; 0/1); Comparable control group (0/1); Allocation of trainees not concealed (1/2); Study retention unlikely source of bias (1/1); No blinded assessment (0/1).
| A: 13.5 B: 13.5
M: 13.5 | A: 2 B: 2
M: 2 | 2 |
| | Music education Wind instrument | To determine whether experienced teachers and inexperienced pre-service teachers have different effects on trainee progress, trainee performance and apply different teaching behaviors when teaching to play a wind instrument. | Single group post test (for the results relevant to our study)
Setting: Trainees selected by the teachers were trained. Training was recorded on video. Data collection: Trainee performance was analyzed by independent experts on a 5-point Likert scale. Trainees were then ranked into the 33% best, intermediate and lowest performing trainees. Teacher and trainee behaviors were analyzed by the researchers with an observational instrument. | Trainees whose performance was classified higher (independent observer assessment: low, intermediate, high, calculated according to a 1 item 5-point rating scale score given by independent observers {A}) talked more during training (scoring on earlier developed observational instrument by researchers {B}). No significant pair-wise effects; no significant correlation. | A: MERSQI: Single group post test (1/3); Three institutions or more (1.5/1.5); Sampling response rate ≥ 75% (1.5/1.5); Objective data assessment (3/3); No validity evidence established (incorrect method for internal validity applied) (0/3); Data analysis beyond descriptive (2/2); Data analysis appropriate (1/1); Outcomes measuring skills acquisition (1.5/3). NOS-E: No representative group of participants (0/1); No comparable group (no separate comparison group; no intervention: 0/1); Non-comparable comparison group (no separate comparison group; no intervention: 0/1); Study retention unlikely source of bias (1/1); Blinded assessment (1/1). B: MERSQI: Single group post test (1/3); Three institutions or more (1.5/1.5); Sampling response rate ≥ 75% (1.5/1.5); Objective data assessment (3/3); Content validity established, no internal validity (incorrect method applied) and relationships to other variables (1/3); Data analysis beyond descriptive (2/2); Data analysis appropriate (1/1); Outcomes measuring skills acquisition (1.5/3). NOS-E: No representative group of participants (0/1); No comparable group (no separate comparison group; no intervention: 0/1); Non-comparable comparison group (no separate comparison group; no intervention: 0/1); Study retention unlikely source of bias (1/1); No blinded assessment (0/1). | A: 11.5 B: 12.5
M: 12 | A: 2 B: 1
M: 1.5 | 2 |
| | Critical medical care education Placing a subclavian central venous catheter (s-CVC) | To develop a procedural objective structured teaching exercise (PrOSTE) and analyze procedural teaching behavior which improves trainee performance. | Single group post test
Setting: A standardized trainee instructed to show challenging learning behavior was trained in s-CVC on a simulator. Consequently, in a following training session, groups of three novice trainees were trained in s-CVC on a simulator. Data collection: A standardized trainee rated the teaching skills of each teacher on a questionnaire. One of the researchers assessed the novice trainees’ s-CVC skills on a checklist. | Higher ratings of a standardized trainee on 3 items (26-item trainee PrOSTE questionnaire; 1-5 {A}) related to higher trainee performance in a consequent training session of novice trainees (assessment by trained observer (researcher); S-CVC 34 item questionnaire; 1-5 {B}). The opposite was found for 1 item. No overall correlation between A and B was found.
| A/B: MERSQI: Single group pre test post test (1/3); One institution (0.5/1.5); Sampling response rate ≥ 75% (1.5/1.5); Objective data assessment (3/3); Content validity established, no internal validity and relationships to other variables (1/3); Data analysis beyond descriptive (2/2); Data analysis appropriate (1/1); Outcomes measuring skills acquisition (1.5/3). NOS-E: No representative group of participants (sampling not described: 0/1); No comparable group (no separate comparison group; no intervention: 0/1); Non-comparable comparison group (no separate comparison group; no intervention: 0/1); Study retention unlikely source of bias (1/1); No blinded assessment (blinding not described; 0/1). | A: 11.5 B: 11.5
M: 11.5 | A: 1 B: 1
M: 1 | 2 |
| | Sports education Golf putt | To analyze and compare the effects of externally and internally focused instruction on learning the golf putt in trainees with no experience. | Randomized controlled trial
Setting: Half the trainees received externally and half received internally focused feedback before each set of training trials. One of the researchers provided all instructions for each trainee during training. During retention no instructions were provided. Data collection: One of the researchers scored the trainees’ accuracy scores based on hitting target areas. | Trainees with externally focused instruction outperformed trainees with internally focused instruction on accuracy (hitting target areas with predetermined points, given by one of the researchers; 0-5 {A}) during training and retention. | A: MERSQI: Randomized controlled trial (3/3); One institution (0.5/1.5); Sampling response rate ≥ 75% (1.5/1.5); Objective data assessment (3/3); Validity evidence N/A; Data analysis beyond descriptive (2/2); Data analysis appropriate (1/1); Outcomes measuring skills acquisition (1.5/3). NOS-E: No representative group of participants (sampling not described; 0/1); Comparable control group (1/1); Allocation of trainees not concealed (1/2); Study retention unlikely source of bias (1/1); No blinded assessment (0/1). | A: 15* | A: 3 | 2 |
| | Sports education Exp. 1 Volleyball serve | To analyze and compare the effects of externally and internally focused feedback on learning the volleyball serve in novice and advanced trainees. | 4 group cohort study
Setting: Half of the novice trainees received externally and half of the novice trainees received internally focused feedback. The same conditions were applied for the advanced trainees. All feedback was standardized, neutrally formulated and provided during training after every fifth trial (50 in total) by one of the researchers. During the retention test no feedback was provided. Data collection: Based on video recordings two independent observers gave movement quality scores according to guidelines. | Trainees with externally focused feedback after each set of trials during training outperformed trainees with internally focused feedback on accuracy (hitting target areas with predetermined points, given by one of the researchers; 0-4 {A}) during training and retention. Effects were independent of trainee experience. Trainees with externally focused feedback outperformed trainees with internally focused feedback on movement quality (independent observer scoring on observational instrument based on guidelines; 0-15 {B}) during training, regardless experience. During retention, the advanced externally feedback group outperformed the advanced internally feedback group. The novice internally feedback group outperformed the novice externally feedback group. | A: MERSQI: Cohort study (2/3); Two institutions (1/1.5); Sampling response rate ≥ 75% (1.5/1.5); Objective data assessment (3/3); Validity evidence N/A; Data analysis beyond descriptive (2/2); Data analysis appropriate (1/1); Outcomes measuring skills acquisition (1.5/3). NOS-E: No representative group of participants (sampling not described; 0/1); Comparable control group (1/1); Allocation of trainees not concealed (1/2); Study retention unlikely source of bias (1/1); No blinded assessment (0/1). B: MERSQI: Cohort study (2/3); Two institutions (1/1.5); Sampling response rate ≥ 75% (1.5/1.5); Objective data assessment (3/3); Content and internal validity established, no relationships to other variables (2/3); Data analysis beyond descriptive (2/2); Data analysis appropriate (1/1); Outcomes measuring skills acquisition (1.5/3). NOS-E: No representative group of participants (sampling not described; 0/1); Comparable control group (1/1); Allocation of trainees not concealed (1/2); Study retention unlikely source of bias (1/1); Blinded assessment (1/1). | A: 14.4* B: 14
M: 14.2 | A: 3 B: 4
M: 3.5 | 2 |
| | Sports education Exp. 2 Soccer pass | To analyze and compare the effects of externally and internally focused feedback on learning the soccer pass with intermittent or constant feedback. | Randomized controlled trial
Setting: Trainees all had experience, received either internally or externally focused feedback, either constant (after every trial) or intermittent (after every third trial; 30 in total). All feedback was standardized, neutrally formulated and provided by one of the researchers. During retention no feedback was provided. Data collection: Accuracy scores were noted by one of the researchers based on target areas. | The externally focused feedback group outperformed the internally focused feedback group on accuracy (hitting target areas with predetermined points, given by one of the researchers; 0-3 {A}) during training and retention, regardless feedback intensity. The intermittent internally focused feedback group outperformed the constant internally focused feedback group on accuracy during training and retention. The constant externally focused feedback group outperformed the intermittent externally focused feedback group. | A: MERSQI: Randomized controlled trial (3/3); One institution (0.5/1.5); Sampling response rate ≥ 75% (1.5/1.5); Objective data assessment (3/3); Validity evidence N/A; Data analysis beyond descriptive (2/2); Data analysis appropriate (1/1); Outcomes measuring skills acquisition (1.5/3). NOS-E: No representative group of participants (sampling not described; 0/1); Comparable control group (1/1); Allocation of trainees not concealed (1/2); Study retention unlikely source of bias (1/1); No blinded assessment (0/1). | A: 15* | A: 3 | 2 |