Exam Development and Scoring

NCCPA’s exam questions are developed by committees comprising PAs and physicians selected based on both their item writing skills, experience and references as well as demographic characteristics (i.e., practice specialty, geographic region, practice setting, etc.). The test committee members each independently write a certain number of test questions or items, referencing each to a recently published textbook (not journal articles). Each item then goes through an intense review by content experts and medical editors from which only some items emerge for pre-testing. Every NCCPA exam includes both scored and pre-test items, and examinees have no way of distinguishing between the two. This allows NCCPA to collect important statistics about how the pre-test items perform on the exam, which informs the final decision about whether a particular question meets the standards for inclusion as a scored item on future PANCE or PANRE exams.

Pathway II exams are developed in much the same way as PANCE and PANRE exams. However, Pathway II questions are not pre-tested due to the nature of that exam. Rather, after a preliminary analysis of each Pathway II administration’s results, statistical analyses are used to identify items that appear to have been problematic or even flawed. Through this validation process, content experts review those items to determine whether the answers had been keyed incorrectly in the scoring system or whether the item itself was flawed in some way. Also, from time to time Pathway II examinees will contact NCCPA with questions or concerns about particular exam items, which are also reviewed during the validation process. When the content experts identify a flawed item, it is removed from the group of scored items and is not included in the scoring process.

When NCCPA exams are scored, candidates are initially awarded 1 point for every correct answer and 0 points for incorrect answers to produce a raw score. After examinees’ raw scores have been computed by two independent computer systems to ensure accuracy, the scored response records for PANCE and PANRE examinees are entered into a maximum likelihood estimation procedure, a sophisticated, mathematically-based procedure that uses the difficulties of all the scored items in the form taken by an individual examinee as well as the number of correct responses to calculate that examinee’s proficiency measure. This calculation is based on the Rasch model and equates the scores, compensating for minor differences in difficulty across different versions of the exam. Thus, in the end, all proficiency measures are calculated as if everyone took the same exam. (That step is not necessary for Pathway II since all examinees in a given administration take the same exam.)

Finally, the proficiency measure is converted to a scaled score so that results can be compared over time and among different groups of examinees. The scale is based on the performance of a reference group (some particular group of examinees who took the exam in the past) whose scores were scaled so that the average proficiency measure was assigned a scaled score of 500 and the standard deviation was established at 100. The vast majority of scores fall between 200 and 800. More details on the reference group for each exam and the calculation of scores will be provided in the form of Performance Interpretation Guidelines published with your exam results.

We do not publish the percent correct level necessary to pass our examinations any more. Given that we have multiple test forms this information would not be accurate since some test forms, while built to be exactly the same, are slightly different in their difficulty. Therefore we convert the percent correct to a scaled score and report scores and the passing standard on that scale.