The bubble charts for the nursing school test, used in the first 20 posts, and for a general studies biology test are strikingly different. The nursing test had a cut score of 75%, was right count scored, and only three of the 24 students failed. The biology test had a cut score of 60%, was knowledge and judgment scored, and only one of the 24 students passed.
The active starting score on the nursing test was set at zero with one point for right and zero for omit and wrong. The active starting score for the biology test was set at 50% (knowledge, and the judgment to use that knowledge, had equal value) with one point for right, ½ point for omit, and zero for wrong. (Partial credit analysis used 2, 1, and 0.)
The averages scores were 84% for nursing and 50% for biology. Nursing students were preparing for a licensure test. Remedial course biology students only needed credit for the course. (It is customary for these freshman students to take this first test without studying to determine if attending lecture will suffice, a case of high school lag.)
The overfit students (1 and 8, for example) tended to omit (good judgment to not make a wrong mark) more than other students. The underfit students (17 and 24, for example) tended to not use good judgment (GJ) as often as other students.
Winsteps flagged four of the six omits on item 24 as unexpected. Rasch model analysis provides an indication of students omitting when the odds were more in favor of getting a right answer than a wrong answer, 4/576, or less than 1% of the time. This rate is less than the average marking error on paper tests.
Only two students, with test scores of 56.3% and 64.6%, are predicted to pass the biology course on PUP Table 3a. Neither student earned scores of 70% or higher (cut score plus one letter grade when using right count scoring). Instead both students demonstrated their ability to use what they knew with quality scores of 100% and 73%.
Ministep estimates the person ability of 21000053 (21) and 25000059 (25) at 0.17 measures. PUP test scores for both are 56.3%. Student 21 may pass because of a quality score of 100%. Student 25 is predicted to fail because of a quality score of only 60%. The quality score was used to break ties in the annual NWMSU Science Olympiad.
PUP Table 3 EGMD makes a selection more interesting. This table of student judgment is unique to Knowledge and Judgment Scoring. It guides students along the path from passive pupil to self-motivated high achiever (average quality score of 90%, always above 80%). Students 21 and 25 differ markedly among the four categories of items. Student 21 unexpectedly marked a misconception right. Student 25 marked three of the four misconceptions wrong. Student 21 omitted all three discriminating items, but student 25 marked all three right. These descriptive data, at the individual level, can yield many stories with unknown reliability. Reliability comes with repeated performance.
Partial credit Rasch model analysis and Knowledge and Judgment Scoring together produce a more meaningful and easy to use classroom and proficiency report than either alone. Caveat: This small sample size was used for illustrative purposes.
No comments:
Post a Comment